all lists on lists.proxmox.com
 help / color / mirror / Atom feed
* [pbs-devel] [PATCH many v4 00/43] integrate notification system
@ 2024-04-22 12:37 Lukas Wagner
  2024-04-22 12:37 ` [pbs-devel] [PATCH proxmox v4 01/43] notify: expose `config` module Lukas Wagner
                   ` (44 more replies)
  0 siblings, 45 replies; 51+ messages in thread
From: Lukas Wagner @ 2024-04-22 12:37 UTC (permalink / raw)
  To: pbs-devel

These patches integrate the notification system which was introduced in Proxmox VE
in 8.1 into Proxmox Backup Server.

Some highlights/noteworthy details from the series:
  - notification template files are installed in /usr/share/proxmox-backup/templates/default
    and are rendered when a notification is sent
  - since sending notifications needs to be done from a privileged context
    (to read protected passwords/tokens from the priv config file),
    we queue notifications to be sent in /var/lib/proxmox-backup/notifications
    and periodically send any queued notifications via a worker in the
    privileged process
  - The API endpoint paths are prefixed with /config/notifications
  - API endpoints which read/modify notification system config require
    Sys.Audit (read) or Sys.Modify (modify/test endpoint) permissions
    on /system/notifications

  - tape-{backup,restore} settings and datastore options now have a
    'notification-mode' parameter, which allows to choose between the
    'legacy' behavior (sendmail to a selected user's email address) and
    the event-based notification system. If the parameter is not set, we
    default to the legacy behavior in order to keep existing behavior
    as is for now. For new datastores/tape backup jobs created
    interactively from the UI, we set the parameter to 'notification-system'
    and opt in into the new system by default. For the CLI/API we don't
    so that we don't cause problems with any automations/scripts -
    there I'd change the default to the new system with the next major release

Rough edges:
  - Datastore option view in UI could be improved. When 'notification-mode' is
    set to 'notification-system', we should indicate that 'mailto-user' and the
    other notification settings have no effect. Already did that in the edit
    window, but in the grid panel I did not find a quick way to do that.

Prerequisites:

  - This patch series requires the patches for 'proxmox' from
  https://lists.proxmox.com/pipermail/pve-devel/2024-April/063493.html
  to be applied first

  - The dependency for proxmox-widget-toolkit should be bumped to at least
  4.1.4 (we need: "utils: add mechanism to add and override translatable notification event
  descriptions in the product specific UIs")

Changes since v3:
  - Pull in UI changes from 'notification metadata matching improvments' as
    component overrides - this can and should be removed once the
    widget toolkit patches are applied.
  - no other changes apart from that

Changes since v2:
  - consolidate verify/prune/sync/tape-job to job-id
  - add a note to docs that package update notifications
    are only sent by nodes with an active subscription

Changes since v1:
  - proxmox_notify: Change default-matcher to exclude 'prune success' notifications
  - proxmox_notify: Move sendmail/forward functions from proxmox_sys
  - proxmox_notify: Improve docs for API types
  - proxmox-widget-toolkit: allow to override default mail author
  - Add documentation, man pages, etc. (mostly copied from PVE, some
    stylistic touchups, some rephrasing)
  - Add additional matchable metadata fields, allowing to match on
    individual jobs:
    - verification-job
    - prune-job
    - sync-job
    - tape-job
  - Sort queued notifications by timestamp before sending
  - Fix missing hostname in package-update notification
  - Change default mail author in the UI from Proxmox VE to Proxmox Backup Server ($hostname)

proxmox:

Lukas Wagner (5):
  notify: expose `config` module
  notify: use std::sync::OnceCell instead of lazy_static!
  notify: pbs-context: exclude successful prunes in default matcher
  notify: endpoints: matcher: improve descriptions for API types
  notify: add getter for notification timestamp

 proxmox-notify/Cargo.toml                |  1 -
 proxmox-notify/src/config.rs             | 23 ++++++++++++-------
 proxmox-notify/src/context/pbs.rs        |  5 ++++-
 proxmox-notify/src/endpoints/gotify.rs   |  6 ++---
 proxmox-notify/src/endpoints/sendmail.rs | 15 ++++++++-----
 proxmox-notify/src/endpoints/smtp.rs     | 28 +++++++++++++++---------
 proxmox-notify/src/lib.rs                | 11 ++++++----
 proxmox-notify/src/matcher.rs            | 18 +++++++--------
 8 files changed, 66 insertions(+), 41 deletions(-)


proxmox-widget-toolkit:

Lukas Wagner (1):
  sendmail: smtp: allow to overide default mail author

 src/Schema.js                  | 9 +++++++++
 src/panel/SendmailEditPanel.js | 2 +-
 src/panel/SmtpEditPanel.js     | 2 +-
 src/window/EndpointEditBase.js | 1 +
 4 files changed, 12 insertions(+), 2 deletions(-)


proxmox-backup:

Lukas Wagner (37):
  pbs-config: add module for loading notification config
  server: rename email_notifications module to notifications
  notifications: allow sending notifications via proxmox_notify
  buildsys: install templates for test notifications
  pbs-config: acl: add /system/notifications as known ACL path
  api: add endpoints for querying/testing notification targets
  api: add endpoints for notification matchers
  api: add endpoints for sendmail targets
  api: add endpoints for smtp targets
  api: add endpoints for gotify targets
  api: add endpoints for querying known notification values/fields
  api-types: api: datatore: add notification-mode parameter
  api-types: api: tape: add notification-mode parameter
  server: notifications: send GC notifications via notification system
  server: notifications: send prune notifications via notification
    system
  server: notifications: send verify notifications via notification
    system
  server: notifications: send sync notifications via notification system
  server: notifications: send update notifications via notification
    system
  server: notifications: send acme notifications via notification system
  server: notifications: send tape notifications via notification system
  ui: add notification config panel
  ui: tape backup job: add selector for notification-mode
  ui: tape backup: add selector for 'notification-mode'
  ui: tape restore: add 'notification-mode' parameter
  ui: datastore options: add 'notification-mode' parameter
  ui: utils: add overrides for known notification metadata fields/values
  ui: datastore edit: make new stores use notification system by default
  ui: permissions paths: add /system/notifications to combobox
  proxmox-backup-manager: add CLI for notification targets
  proxmox-backup-manager: add CLI for notification matchers
  proxmox-backup-manager: add CLI for gotify endpoints
  proxmox-backup-manager: add CLI for sendmail endpoints
  proxmox-backup-manager: add CLI for SMTP endpoints
  docgen: generate synopsis for notifications{-priv,}.cfg
  docs: add documentation for notification system
  ui: util: override default mail author for sendmail/smtp targets
  ui: notifications: pull in UX improvements for match rules creation

 Cargo.toml                                    |    3 +
 Makefile                                      |    5 +-
 debian/proxmox-backup-server.install          |   31 +
 docs/Makefile                                 |    6 +-
 docs/conf.py                                  |    2 +
 docs/config/notifications-priv/format.rst     |    1 +
 docs/config/notifications-priv/man5.rst       |   24 +
 docs/config/notifications/format.rst          |   28 +
 docs/config/notifications/man5.rst            |   24 +
 docs/configuration-files.rst                  |   30 +
 docs/index.rst                                |    1 +
 docs/notifications.rst                        |  213 ++++
 pbs-api-types/src/datastore.rs                |   22 +
 pbs-api-types/src/jobs.rs                     |    8 +-
 pbs-config/Cargo.toml                         |    1 +
 pbs-config/src/acl.rs                         |    3 +-
 pbs-config/src/lib.rs                         |    1 +
 pbs-config/src/notifications.rs               |   41 +
 src/api2/config/datastore.rs                  |    9 +
 src/api2/config/mod.rs                        |    2 +
 src/api2/config/notifications/gotify.rs       |  190 +++
 src/api2/config/notifications/matchers.rs     |  170 +++
 src/api2/config/notifications/mod.rs          |  210 ++++
 src/api2/config/notifications/sendmail.rs     |  178 +++
 src/api2/config/notifications/smtp.rs         |  191 +++
 src/api2/config/notifications/targets.rs      |   63 +
 src/api2/config/tape_backup_job.rs            |    8 +
 src/api2/pull.rs                              |   10 +-
 src/api2/tape/backup.rs                       |   62 +-
 src/api2/tape/restore.rs                      |   46 +-
 src/bin/docgen.rs                             |    4 +
 src/bin/proxmox-backup-api.rs                 |   11 +
 src/bin/proxmox-backup-manager.rs             |    2 +
 src/bin/proxmox-backup-proxy.rs               |    1 +
 src/bin/proxmox_backup_manager/mod.rs         |    2 +
 .../notifications/gotify.rs                   |   93 ++
 .../notifications/matchers.rs                 |   93 ++
 .../notifications/mod.rs                      |   21 +
 .../notifications/sendmail.rs                 |   94 ++
 .../notifications/smtp.rs                     |   96 ++
 .../notifications/targets.rs                  |   51 +
 src/server/email_notifications.rs             |  763 ------------
 src/server/gc_job.rs                          |   10 +-
 src/server/mod.rs                             |    4 +-
 src/server/notifications.rs                   |  566 +++++++++
 src/server/verify_job.rs                      |   10 +-
 src/tape/drive/mod.rs                         |   22 +-
 src/tape/mod.rs                               |   27 +
 src/tape/pool_writer/mod.rs                   |   11 +-
 templates/Makefile                            |   41 +
 templates/default/acme-err-body.txt.hbs       |    7 +
 templates/default/acme-err-subject.txt.hbs    |    1 +
 templates/default/gc-err-body.txt.hbs         |    8 +
 templates/default/gc-err-subject.txt.hbs      |    1 +
 templates/default/gc-ok-body.txt.hbs          |   23 +
 templates/default/gc-ok-subject.txt.hbs       |    1 +
 .../default/package-updates-body.txt.hbs      |    8 +
 .../default/package-updates-subject.txt.hbs   |    1 +
 templates/default/prune-err-body.txt.hbs      |   10 +
 templates/default/prune-err-subject.txt.hbs   |    1 +
 templates/default/prune-ok-body.txt.hbs       |   10 +
 templates/default/prune-ok-subject.txt.hbs    |    1 +
 templates/default/sync-err-body.txt.hbs       |   14 +
 templates/default/sync-err-subject.txt.hbs    |    5 +
 templates/default/sync-ok-body.txt.hbs        |   14 +
 templates/default/sync-ok-subject.txt.hbs     |    5 +
 .../default/tape-backup-err-body.txt.hbs      |   26 +
 .../default/tape-backup-err-subject.txt.hbs   |    5 +
 templates/default/tape-backup-ok-body.txt.hbs |   27 +
 .../default/tape-backup-ok-subject.txt.hbs    |    5 +
 templates/default/tape-load-body.txt.hbs      |   15 +
 templates/default/tape-load-subject.txt.hbs   |    1 +
 templates/default/test-body.html.hbs          |    1 +
 templates/default/test-body.txt.hbs           |    1 +
 templates/default/test-subject.txt.hbs        |    1 +
 templates/default/verify-err-body.txt.hbs     |   14 +
 templates/default/verify-err-subject.txt.hbs  |    1 +
 templates/default/verify-ok-body.txt.hbs      |   10 +
 templates/default/verify-ok-subject.txt.hbs   |    1 +
 www/Makefile                                  |    2 +
 www/NavigationTree.js                         |    6 +
 www/OnlineHelpInfo.js                         |   24 +
 www/Utils.js                                  |   39 +
 www/config/NotificationConfigView.js          |   11 +
 www/datastore/OptionView.js                   |   15 +
 www/form/PermissionPathSelector.js            |    1 +
 www/tape/window/TapeBackup.js                 |   25 +
 www/tape/window/TapeBackupJob.js              |   24 +
 www/tape/window/TapeRestore.js                |   17 +
 www/window/DataStoreEdit.js                   |   13 +
 www/window/NotificationMatcherOverride.js     | 1105 +++++++++++++++++
 www/window/NotifyOptions.js                   |   38 +
 92 files changed, 4174 insertions(+), 869 deletions(-)
 create mode 100644 docs/config/notifications-priv/format.rst
 create mode 100644 docs/config/notifications-priv/man5.rst
 create mode 100644 docs/config/notifications/format.rst
 create mode 100644 docs/config/notifications/man5.rst
 create mode 100644 docs/notifications.rst
 create mode 100644 pbs-config/src/notifications.rs
 create mode 100644 src/api2/config/notifications/gotify.rs
 create mode 100644 src/api2/config/notifications/matchers.rs
 create mode 100644 src/api2/config/notifications/mod.rs
 create mode 100644 src/api2/config/notifications/sendmail.rs
 create mode 100644 src/api2/config/notifications/smtp.rs
 create mode 100644 src/api2/config/notifications/targets.rs
 create mode 100644 src/bin/proxmox_backup_manager/notifications/gotify.rs
 create mode 100644 src/bin/proxmox_backup_manager/notifications/matchers.rs
 create mode 100644 src/bin/proxmox_backup_manager/notifications/mod.rs
 create mode 100644 src/bin/proxmox_backup_manager/notifications/sendmail.rs
 create mode 100644 src/bin/proxmox_backup_manager/notifications/smtp.rs
 create mode 100644 src/bin/proxmox_backup_manager/notifications/targets.rs
 delete mode 100644 src/server/email_notifications.rs
 create mode 100644 src/server/notifications.rs
 create mode 100644 templates/Makefile
 create mode 100644 templates/default/acme-err-body.txt.hbs
 create mode 100644 templates/default/acme-err-subject.txt.hbs
 create mode 100644 templates/default/gc-err-body.txt.hbs
 create mode 100644 templates/default/gc-err-subject.txt.hbs
 create mode 100644 templates/default/gc-ok-body.txt.hbs
 create mode 100644 templates/default/gc-ok-subject.txt.hbs
 create mode 100644 templates/default/package-updates-body.txt.hbs
 create mode 100644 templates/default/package-updates-subject.txt.hbs
 create mode 100644 templates/default/prune-err-body.txt.hbs
 create mode 100644 templates/default/prune-err-subject.txt.hbs
 create mode 100644 templates/default/prune-ok-body.txt.hbs
 create mode 100644 templates/default/prune-ok-subject.txt.hbs
 create mode 100644 templates/default/sync-err-body.txt.hbs
 create mode 100644 templates/default/sync-err-subject.txt.hbs
 create mode 100644 templates/default/sync-ok-body.txt.hbs
 create mode 100644 templates/default/sync-ok-subject.txt.hbs
 create mode 100644 templates/default/tape-backup-err-body.txt.hbs
 create mode 100644 templates/default/tape-backup-err-subject.txt.hbs
 create mode 100644 templates/default/tape-backup-ok-body.txt.hbs
 create mode 100644 templates/default/tape-backup-ok-subject.txt.hbs
 create mode 100644 templates/default/tape-load-body.txt.hbs
 create mode 100644 templates/default/tape-load-subject.txt.hbs
 create mode 100644 templates/default/test-body.html.hbs
 create mode 100644 templates/default/test-body.txt.hbs
 create mode 100644 templates/default/test-subject.txt.hbs
 create mode 100644 templates/default/verify-err-body.txt.hbs
 create mode 100644 templates/default/verify-err-subject.txt.hbs
 create mode 100644 templates/default/verify-ok-body.txt.hbs
 create mode 100644 templates/default/verify-ok-subject.txt.hbs
 create mode 100644 www/config/NotificationConfigView.js
 create mode 100644 www/window/NotificationMatcherOverride.js


Summary over all repositories:
  104 files changed, 4252 insertions(+), 912 deletions(-)

-- 
Generated by git-murpp 0.7.1


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel


^ permalink raw reply	[flat|nested] 51+ messages in thread

end of thread, other threads:[~2024-04-23 11:44 UTC | newest]

Thread overview: 51+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-04-22 12:37 [pbs-devel] [PATCH many v4 00/43] integrate notification system Lukas Wagner
2024-04-22 12:37 ` [pbs-devel] [PATCH proxmox v4 01/43] notify: expose `config` module Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox v4 02/43] notify: use std::sync::OnceCell instead of lazy_static! Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox v4 03/43] notify: pbs-context: exclude successful prunes in default matcher Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox v4 04/43] notify: endpoints: matcher: improve descriptions for API types Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox v4 05/43] notify: add getter for notification timestamp Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH widget-toolkit v4 06/43] sendmail: smtp: allow to overide default mail author Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 07/43] pbs-config: add module for loading notification config Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 08/43] server: rename email_notifications module to notifications Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 09/43] notifications: allow sending notifications via proxmox_notify Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 10/43] buildsys: install templates for test notifications Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 11/43] pbs-config: acl: add /system/notifications as known ACL path Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 12/43] api: add endpoints for querying/testing notification targets Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 13/43] api: add endpoints for notification matchers Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 14/43] api: add endpoints for sendmail targets Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 15/43] api: add endpoints for smtp targets Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 16/43] api: add endpoints for gotify targets Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 17/43] api: add endpoints for querying known notification values/fields Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 18/43] api-types: api: datatore: add notification-mode parameter Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 19/43] api-types: api: tape: " Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 20/43] server: notifications: send GC notifications via notification system Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 21/43] server: notifications: send prune " Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 22/43] server: notifications: send verify " Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 23/43] server: notifications: send sync " Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 24/43] server: notifications: send update " Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 25/43] server: notifications: send acme " Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 26/43] server: notifications: send tape " Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 27/43] ui: add notification config panel Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 28/43] ui: tape backup job: add selector for notification-mode Lukas Wagner
2024-04-22 14:32   ` Dominik Csapak
2024-04-23  8:14     ` Lukas Wagner
2024-04-23  8:16       ` Dominik Csapak
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 29/43] ui: tape backup: add selector for 'notification-mode' Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 30/43] ui: tape restore: add 'notification-mode' parameter Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 31/43] ui: datastore options: " Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 32/43] ui: utils: add overrides for known notification metadata fields/values Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 33/43] ui: datastore edit: make new stores use notification system by default Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 34/43] ui: permissions paths: add /system/notifications to combobox Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 35/43] proxmox-backup-manager: add CLI for notification targets Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 36/43] proxmox-backup-manager: add CLI for notification matchers Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 37/43] proxmox-backup-manager: add CLI for gotify endpoints Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 38/43] proxmox-backup-manager: add CLI for sendmail endpoints Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 39/43] proxmox-backup-manager: add CLI for SMTP endpoints Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 40/43] docgen: generate synopsis for notifications{-priv, }.cfg Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 41/43] docs: add documentation for notification system Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 42/43] ui: util: override default mail author for sendmail/smtp targets Lukas Wagner
2024-04-22 12:38 ` [pbs-devel] [PATCH proxmox-backup v4 43/43] ui: notifications: pull in UX improvements for match rules creation Lukas Wagner
2024-04-22 13:58 ` [pbs-devel] [PATCH many v4 00/43] integrate notification system Maximiliano Sandoval
2024-04-23 11:44   ` Lukas Wagner
2024-04-22 14:24 ` Fabian Grünbichler
2024-04-23 11:13   ` Lukas Wagner

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal