public inbox for pbs-devel@lists.proxmox.com
 help / color / mirror / Atom feed
* [pbs-devel] [PATCH many v2 00/43] integrate notification system
@ 2024-04-17 14:34 Lukas Wagner
  2024-04-17 14:34 ` [pbs-devel] [PATCH proxmox v2 01/43] notify: expose `config` module Lukas Wagner
                   ` (42 more replies)
  0 siblings, 43 replies; 44+ messages in thread
From: Lukas Wagner @ 2024-04-17 14:34 UTC (permalink / raw)
  To: pbs-devel

These patches integrate the notification system which was introduced in Proxmox VE
in 8.1 into Proxmox Backup Server.

Some highlights/noteworthy details from the series:
  - notification template files are installed in /usr/share/proxmox-backup/templates/default
    and are rendered when a notification is sent
  - since sending notifications needs to be done from a privileged context
    (to read protected passwords/tokens from the priv config file),
    we queue notifications to be sent in /var/lib/proxmox-backup/notifications
    and periodically send any queued notifications via a worker in the
    privileged process
  - The API endpoint paths are prefixed with /config/notifications
  - API endpoints which read/modify notification system config require
    Sys.Audit (read) or Sys.Modify (modify/test endpoint) permissions
    on /system/notifications

  - tape-{backup,restore} settings and datastore options now have a
    'notification-mode' parameter, which allows to choose between the
    'legacy' behavior (sendmail to a selected user's email address) and
    the event-based notification system. If the parameter is not set, we
    default to the legacy behavior in order to keep existing behavior
    as is for now. For new datastores/tape backup jobs created
    interactively from the UI, we set the parameter to 'notification-system'
    and opt in into the new system by default. For the CLI/API we don't
    so that we don't cause problems with any automations/scripts -
    there I'd change the default to the new system with the next major release

Rough edges:
  - Datastore option view in UI could be improved. When 'notification-mode' is
    set to 'notification-system', we should indicate that 'mailto-user' and the
    other notification settings have no effect. Already did that in the edit
    window, but in the grid panel I did not find a quick way to do that.

Prerequisites:

  - This patch series requires the patches for 'proxmox' from
  https://lists.proxmox.com/pipermail/pve-devel/2024-April/062708.html
  to be applied first

  - The dependency for proxmox-widget-toolkit should be bumped to at least
  4.1.4 (we need: "utils: add mechanism to add and override translatable notification event
  descriptions in the product specific UIs")
  Also, while not strictly needed, the patches for widget-toolkit from
  https://lists.proxmox.com/pipermail/pve-devel/2024-February/061992.html
  make creating matchers much easier - the required API endpoints were
  already implemented by this series.

Changes since v1:
  - proxmox_notify: Change default-matcher to exclude 'prune success' notifications
  - proxmox_notify: Move sendmail/forward functions from proxmox_sys
  - proxmox_notify: Improve docs for API types
  - proxmox-widget-toolkit: allow to override default mail author
  - Add documentation, man pages, etc. (mostly copied from PVE, some
    stylistic touchups, some rephrasing)
  - Add additional matchable metadata fields, allowing to match on
    individual jobs:
    - verification-job
    - prune-job
    - sync-job
    - tape-job
  - Sort queued notifications by timestamp before sending
  - Fix missing hostname in package-update notification
  - Change default mail author in the UI from Proxmox VE to Proxmox Backup Server ($hostname)

proxmox:

Lukas Wagner (6):
  notify: expose `config` module
  notify: use std::sync::OnceCell instead of lazy_static!
  notify: pbs-context: exclude successful prunes in default matcher
  notify: endpoints: matcher: improve descriptions for API types
  notify: sys: move 'email' module from proxmox_sys to proxmox_notify
  notify: add getter for notification timestamp

 proxmox-notify/Cargo.toml                     |  4 +--
 proxmox-notify/src/config.rs                  | 23 +++++++++------
 proxmox-notify/src/context/pbs.rs             |  5 +++-
 proxmox-notify/src/endpoints/common/mod.rs    |  3 ++
 .../src/endpoints/common/sendmail_impl.rs     | 10 ++++---
 proxmox-notify/src/endpoints/gotify.rs        |  6 ++--
 proxmox-notify/src/endpoints/sendmail.rs      | 20 ++++++++-----
 proxmox-notify/src/endpoints/smtp.rs          | 28 ++++++++++++-------
 proxmox-notify/src/lib.rs                     | 11 +++++---
 proxmox-notify/src/matcher.rs                 | 18 ++++++------
 proxmox-sys/src/lib.rs                        |  1 -
 11 files changed, 80 insertions(+), 49 deletions(-)
 rename proxmox-sys/src/email.rs => proxmox-notify/src/endpoints/common/sendmail_impl.rs (96%)


proxmox-widget-toolkit:

Lukas Wagner (1):
  sendmail: smtp: allow to overide default mail author

 src/Schema.js                  | 9 +++++++++
 src/panel/SendmailEditPanel.js | 2 +-
 src/panel/SmtpEditPanel.js     | 2 +-
 src/window/EndpointEditBase.js | 1 +
 4 files changed, 12 insertions(+), 2 deletions(-)


proxmox-backup:

Lukas Wagner (36):
  pbs-config: add module for loading notification config
  server: rename email_notifications module to notifications
  notifications: allow sending notifications via proxmox_notify
  buildsys: install templates for test notifications
  pbs-config: acl: add /system/notifications as known ACL path
  api: add endpoints for querying/testing notification targets
  api: add endpoints for notification matchers
  api: add endpoints for sendmail targets
  api: add endpoints for smtp targets
  api: add endpoints for gotify targets
  api: add endpoints for querying known notification values/fields
  api-types: api: datatore: add notification-mode parameter
  api-types: api: tape: add notification-mode parameter
  server: notifications: send GC notifications via notification system
  server: notifications: send prune notifications via notification
    system
  server: notifications: send verify notifications via notification
    system
  server: notifications: send sync notifications via notification system
  server: notifications: send update notifications via notification
    system
  server: notifications: send acme notifications via notification system
  server: notifications: send tape notifications via notification system
  ui: add notification config panel
  ui: tape backup job: add selector for notification-mode
  ui: tape backup: add selector for 'notification-mode'
  ui: tape restore: add 'notification-mode' parameter
  ui: datastore options: add 'notification-mode' parameter
  ui: utils: add overrides for known notification metadata fields/values
  ui: datastore edit: make new stores use notification system by default
  ui: permissions paths: add /system/notifications to combobox
  proxmox-backup-manager: add CLI for notification targets
  proxmox-backup-manager: add CLI for notification matchers
  proxmox-backup-manager: add CLI for gotify endpoints
  proxmox-backup-manager: add CLI for sendmail endpoints
  proxmox-backup-manager: add CLI for SMTP endpoints
  docgen: generate synopsis for notifications{-priv,}.cfg
  docs: add documentation for notification system
  ui: util: override default mail author for sendmail/smtp targets

 Cargo.toml                                    |   3 +
 Makefile                                      |   5 +-
 debian/proxmox-backup-server.install          |  31 +
 docs/Makefile                                 |   6 +-
 docs/conf.py                                  |   2 +
 docs/config/notifications-priv/format.rst     |   1 +
 docs/config/notifications-priv/man5.rst       |  24 +
 docs/config/notifications/format.rst          |  28 +
 docs/config/notifications/man5.rst            |  24 +
 docs/configuration-files.rst                  |  30 +
 docs/index.rst                                |   1 +
 docs/notifications.rst                        | 213 +++++
 pbs-api-types/src/datastore.rs                |  22 +
 pbs-api-types/src/jobs.rs                     |   8 +-
 pbs-config/Cargo.toml                         |   1 +
 pbs-config/src/acl.rs                         |   3 +-
 pbs-config/src/lib.rs                         |   1 +
 pbs-config/src/notifications.rs               |  41 +
 src/api2/config/datastore.rs                  |   9 +
 src/api2/config/mod.rs                        |   2 +
 src/api2/config/notifications/gotify.rs       | 190 +++++
 src/api2/config/notifications/matchers.rs     | 170 ++++
 src/api2/config/notifications/mod.rs          | 224 +++++
 src/api2/config/notifications/sendmail.rs     | 178 ++++
 src/api2/config/notifications/smtp.rs         | 191 +++++
 src/api2/config/notifications/targets.rs      |  63 ++
 src/api2/config/tape_backup_job.rs            |   8 +
 src/api2/pull.rs                              |  10 +-
 src/api2/tape/backup.rs                       |  62 +-
 src/api2/tape/restore.rs                      |  46 +-
 src/bin/docgen.rs                             |   4 +
 src/bin/proxmox-backup-api.rs                 |  11 +
 src/bin/proxmox-backup-manager.rs             |   2 +
 src/bin/proxmox-backup-proxy.rs               |   1 +
 src/bin/proxmox_backup_manager/mod.rs         |   2 +
 .../notifications/gotify.rs                   |  93 +++
 .../notifications/matchers.rs                 |  93 +++
 .../notifications/mod.rs                      |  21 +
 .../notifications/sendmail.rs                 |  94 +++
 .../notifications/smtp.rs                     |  96 +++
 .../notifications/targets.rs                  |  51 ++
 src/server/email_notifications.rs             | 763 ------------------
 src/server/gc_job.rs                          |  10 +-
 src/server/mod.rs                             |   4 +-
 src/server/notifications.rs                   | 566 +++++++++++++
 src/server/verify_job.rs                      |  10 +-
 src/tape/drive/mod.rs                         |  22 +-
 src/tape/mod.rs                               |  27 +
 src/tape/pool_writer/mod.rs                   |  11 +-
 templates/Makefile                            |  41 +
 templates/default/acme-err-body.txt.hbs       |   7 +
 templates/default/acme-err-subject.txt.hbs    |   1 +
 templates/default/gc-err-body.txt.hbs         |   8 +
 templates/default/gc-err-subject.txt.hbs      |   1 +
 templates/default/gc-ok-body.txt.hbs          |  23 +
 templates/default/gc-ok-subject.txt.hbs       |   1 +
 .../default/package-updates-body.txt.hbs      |   8 +
 .../default/package-updates-subject.txt.hbs   |   1 +
 templates/default/prune-err-body.txt.hbs      |  10 +
 templates/default/prune-err-subject.txt.hbs   |   1 +
 templates/default/prune-ok-body.txt.hbs       |  10 +
 templates/default/prune-ok-subject.txt.hbs    |   1 +
 templates/default/sync-err-body.txt.hbs       |  14 +
 templates/default/sync-err-subject.txt.hbs    |   5 +
 templates/default/sync-ok-body.txt.hbs        |  14 +
 templates/default/sync-ok-subject.txt.hbs     |   5 +
 .../default/tape-backup-err-body.txt.hbs      |  26 +
 .../default/tape-backup-err-subject.txt.hbs   |   5 +
 templates/default/tape-backup-ok-body.txt.hbs |  27 +
 .../default/tape-backup-ok-subject.txt.hbs    |   5 +
 templates/default/tape-load-body.txt.hbs      |  15 +
 templates/default/tape-load-subject.txt.hbs   |   1 +
 templates/default/test-body.html.hbs          |   1 +
 templates/default/test-body.txt.hbs           |   1 +
 templates/default/test-subject.txt.hbs        |   1 +
 templates/default/verify-err-body.txt.hbs     |  14 +
 templates/default/verify-err-subject.txt.hbs  |   1 +
 templates/default/verify-ok-body.txt.hbs      |  10 +
 templates/default/verify-ok-subject.txt.hbs   |   1 +
 www/Makefile                                  |   1 +
 www/NavigationTree.js                         |   6 +
 www/OnlineHelpInfo.js                         |  24 +
 www/Utils.js                                  |  40 +
 www/config/NotificationConfigView.js          |  11 +
 www/datastore/OptionView.js                   |  15 +
 www/form/PermissionPathSelector.js            |   1 +
 www/tape/window/TapeBackup.js                 |  25 +
 www/tape/window/TapeBackupJob.js              |  24 +
 www/tape/window/TapeRestore.js                |  17 +
 www/window/DataStoreEdit.js                   |  13 +
 www/window/NotifyOptions.js                   |  38 +
 91 files changed, 3083 insertions(+), 869 deletions(-)
 create mode 100644 docs/config/notifications-priv/format.rst
 create mode 100644 docs/config/notifications-priv/man5.rst
 create mode 100644 docs/config/notifications/format.rst
 create mode 100644 docs/config/notifications/man5.rst
 create mode 100644 docs/notifications.rst
 create mode 100644 pbs-config/src/notifications.rs
 create mode 100644 src/api2/config/notifications/gotify.rs
 create mode 100644 src/api2/config/notifications/matchers.rs
 create mode 100644 src/api2/config/notifications/mod.rs
 create mode 100644 src/api2/config/notifications/sendmail.rs
 create mode 100644 src/api2/config/notifications/smtp.rs
 create mode 100644 src/api2/config/notifications/targets.rs
 create mode 100644 src/bin/proxmox_backup_manager/notifications/gotify.rs
 create mode 100644 src/bin/proxmox_backup_manager/notifications/matchers.rs
 create mode 100644 src/bin/proxmox_backup_manager/notifications/mod.rs
 create mode 100644 src/bin/proxmox_backup_manager/notifications/sendmail.rs
 create mode 100644 src/bin/proxmox_backup_manager/notifications/smtp.rs
 create mode 100644 src/bin/proxmox_backup_manager/notifications/targets.rs
 delete mode 100644 src/server/email_notifications.rs
 create mode 100644 src/server/notifications.rs
 create mode 100644 templates/Makefile
 create mode 100644 templates/default/acme-err-body.txt.hbs
 create mode 100644 templates/default/acme-err-subject.txt.hbs
 create mode 100644 templates/default/gc-err-body.txt.hbs
 create mode 100644 templates/default/gc-err-subject.txt.hbs
 create mode 100644 templates/default/gc-ok-body.txt.hbs
 create mode 100644 templates/default/gc-ok-subject.txt.hbs
 create mode 100644 templates/default/package-updates-body.txt.hbs
 create mode 100644 templates/default/package-updates-subject.txt.hbs
 create mode 100644 templates/default/prune-err-body.txt.hbs
 create mode 100644 templates/default/prune-err-subject.txt.hbs
 create mode 100644 templates/default/prune-ok-body.txt.hbs
 create mode 100644 templates/default/prune-ok-subject.txt.hbs
 create mode 100644 templates/default/sync-err-body.txt.hbs
 create mode 100644 templates/default/sync-err-subject.txt.hbs
 create mode 100644 templates/default/sync-ok-body.txt.hbs
 create mode 100644 templates/default/sync-ok-subject.txt.hbs
 create mode 100644 templates/default/tape-backup-err-body.txt.hbs
 create mode 100644 templates/default/tape-backup-err-subject.txt.hbs
 create mode 100644 templates/default/tape-backup-ok-body.txt.hbs
 create mode 100644 templates/default/tape-backup-ok-subject.txt.hbs
 create mode 100644 templates/default/tape-load-body.txt.hbs
 create mode 100644 templates/default/tape-load-subject.txt.hbs
 create mode 100644 templates/default/test-body.html.hbs
 create mode 100644 templates/default/test-body.txt.hbs
 create mode 100644 templates/default/test-subject.txt.hbs
 create mode 100644 templates/default/verify-err-body.txt.hbs
 create mode 100644 templates/default/verify-err-subject.txt.hbs
 create mode 100644 templates/default/verify-ok-body.txt.hbs
 create mode 100644 templates/default/verify-ok-subject.txt.hbs
 create mode 100644 www/config/NotificationConfigView.js


Summary over all repositories:
  106 files changed, 3175 insertions(+), 920 deletions(-)

-- 
Generated by git-murpp 0.7.1


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel


^ permalink raw reply	[flat|nested] 44+ messages in thread

end of thread, other threads:[~2024-04-17 14:45 UTC | newest]

Thread overview: 44+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-04-17 14:34 [pbs-devel] [PATCH many v2 00/43] integrate notification system Lukas Wagner
2024-04-17 14:34 ` [pbs-devel] [PATCH proxmox v2 01/43] notify: expose `config` module Lukas Wagner
2024-04-17 14:34 ` [pbs-devel] [PATCH proxmox v2 02/43] notify: use std::sync::OnceCell instead of lazy_static! Lukas Wagner
2024-04-17 14:34 ` [pbs-devel] [PATCH proxmox v2 03/43] notify: pbs-context: exclude successful prunes in default matcher Lukas Wagner
2024-04-17 14:34 ` [pbs-devel] [PATCH proxmox v2 04/43] notify: endpoints: matcher: improve descriptions for API types Lukas Wagner
2024-04-17 14:34 ` [pbs-devel] [PATCH proxmox v2 05/43] notify: sys: move 'email' module from proxmox_sys to proxmox_notify Lukas Wagner
2024-04-17 14:34 ` [pbs-devel] [PATCH proxmox v2 06/43] notify: add getter for notification timestamp Lukas Wagner
2024-04-17 14:34 ` [pbs-devel] [PATCH widget-toolkit v2 07/43] sendmail: smtp: allow to overide default mail author Lukas Wagner
2024-04-17 14:34 ` [pbs-devel] [PATCH proxmox-backup v2 08/43] pbs-config: add module for loading notification config Lukas Wagner
2024-04-17 14:34 ` [pbs-devel] [PATCH proxmox-backup v2 09/43] server: rename email_notifications module to notifications Lukas Wagner
2024-04-17 14:34 ` [pbs-devel] [PATCH proxmox-backup v2 10/43] notifications: allow sending notifications via proxmox_notify Lukas Wagner
2024-04-17 14:34 ` [pbs-devel] [PATCH proxmox-backup v2 11/43] buildsys: install templates for test notifications Lukas Wagner
2024-04-17 14:34 ` [pbs-devel] [PATCH proxmox-backup v2 12/43] pbs-config: acl: add /system/notifications as known ACL path Lukas Wagner
2024-04-17 14:34 ` [pbs-devel] [PATCH proxmox-backup v2 13/43] api: add endpoints for querying/testing notification targets Lukas Wagner
2024-04-17 14:34 ` [pbs-devel] [PATCH proxmox-backup v2 14/43] api: add endpoints for notification matchers Lukas Wagner
2024-04-17 14:34 ` [pbs-devel] [PATCH proxmox-backup v2 15/43] api: add endpoints for sendmail targets Lukas Wagner
2024-04-17 14:34 ` [pbs-devel] [PATCH proxmox-backup v2 16/43] api: add endpoints for smtp targets Lukas Wagner
2024-04-17 14:34 ` [pbs-devel] [PATCH proxmox-backup v2 17/43] api: add endpoints for gotify targets Lukas Wagner
2024-04-17 14:34 ` [pbs-devel] [PATCH proxmox-backup v2 18/43] api: add endpoints for querying known notification values/fields Lukas Wagner
2024-04-17 14:34 ` [pbs-devel] [PATCH proxmox-backup v2 19/43] api-types: api: datatore: add notification-mode parameter Lukas Wagner
2024-04-17 14:34 ` [pbs-devel] [PATCH proxmox-backup v2 20/43] api-types: api: tape: " Lukas Wagner
2024-04-17 14:34 ` [pbs-devel] [PATCH proxmox-backup v2 21/43] server: notifications: send GC notifications via notification system Lukas Wagner
2024-04-17 14:34 ` [pbs-devel] [PATCH proxmox-backup v2 22/43] server: notifications: send prune " Lukas Wagner
2024-04-17 14:34 ` [pbs-devel] [PATCH proxmox-backup v2 23/43] server: notifications: send verify " Lukas Wagner
2024-04-17 14:34 ` [pbs-devel] [PATCH proxmox-backup v2 24/43] server: notifications: send sync " Lukas Wagner
2024-04-17 14:34 ` [pbs-devel] [PATCH proxmox-backup v2 25/43] server: notifications: send update " Lukas Wagner
2024-04-17 14:34 ` [pbs-devel] [PATCH proxmox-backup v2 26/43] server: notifications: send acme " Lukas Wagner
2024-04-17 14:34 ` [pbs-devel] [PATCH proxmox-backup v2 27/43] server: notifications: send tape " Lukas Wagner
2024-04-17 14:34 ` [pbs-devel] [PATCH proxmox-backup v2 28/43] ui: add notification config panel Lukas Wagner
2024-04-17 14:35 ` [pbs-devel] [PATCH proxmox-backup v2 29/43] ui: tape backup job: add selector for notification-mode Lukas Wagner
2024-04-17 14:35 ` [pbs-devel] [PATCH proxmox-backup v2 30/43] ui: tape backup: add selector for 'notification-mode' Lukas Wagner
2024-04-17 14:35 ` [pbs-devel] [PATCH proxmox-backup v2 31/43] ui: tape restore: add 'notification-mode' parameter Lukas Wagner
2024-04-17 14:35 ` [pbs-devel] [PATCH proxmox-backup v2 32/43] ui: datastore options: " Lukas Wagner
2024-04-17 14:35 ` [pbs-devel] [PATCH proxmox-backup v2 33/43] ui: utils: add overrides for known notification metadata fields/values Lukas Wagner
2024-04-17 14:35 ` [pbs-devel] [PATCH proxmox-backup v2 34/43] ui: datastore edit: make new stores use notification system by default Lukas Wagner
2024-04-17 14:35 ` [pbs-devel] [PATCH proxmox-backup v2 35/43] ui: permissions paths: add /system/notifications to combobox Lukas Wagner
2024-04-17 14:35 ` [pbs-devel] [PATCH proxmox-backup v2 36/43] proxmox-backup-manager: add CLI for notification targets Lukas Wagner
2024-04-17 14:35 ` [pbs-devel] [PATCH proxmox-backup v2 37/43] proxmox-backup-manager: add CLI for notification matchers Lukas Wagner
2024-04-17 14:35 ` [pbs-devel] [PATCH proxmox-backup v2 38/43] proxmox-backup-manager: add CLI for gotify endpoints Lukas Wagner
2024-04-17 14:35 ` [pbs-devel] [PATCH proxmox-backup v2 39/43] proxmox-backup-manager: add CLI for sendmail endpoints Lukas Wagner
2024-04-17 14:35 ` [pbs-devel] [PATCH proxmox-backup v2 40/43] proxmox-backup-manager: add CLI for SMTP endpoints Lukas Wagner
2024-04-17 14:35 ` [pbs-devel] [PATCH proxmox-backup v2 41/43] docgen: generate synopsis for notifications{-priv, }.cfg Lukas Wagner
2024-04-17 14:35 ` [pbs-devel] [PATCH proxmox-backup v2 42/43] docs: add documentation for notification system Lukas Wagner
2024-04-17 14:35 ` [pbs-devel] [PATCH proxmox-backup v2 43/43] ui: util: override default mail author for sendmail/smtp targets Lukas Wagner

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal