all lists on lists.proxmox.com
 help / color / mirror / Atom feed
* [pbs-devel] [PATCH many v5 00/44] integrate notification system
@ 2024-04-23 11:51 Lukas Wagner
  2024-04-23 11:51 ` [pbs-devel] [PATCH proxmox v5 01/44] notify: expose `config` module Lukas Wagner
                   ` (44 more replies)
  0 siblings, 45 replies; 47+ messages in thread
From: Lukas Wagner @ 2024-04-23 11:51 UTC (permalink / raw)
  To: pbs-devel

These patches integrate the notification system which was introduced in Proxmox VE
in 8.1 into Proxmox Backup Server.

Some highlights/noteworthy details from the series:
  - notification template files are installed in /usr/share/proxmox-backup/templates/default
    and are rendered when a notification is sent
  - since sending notifications needs to be done from a privileged context
    (to read protected passwords/tokens from the priv config file),
    we queue notifications to be sent in /var/lib/proxmox-backup/notifications
    and periodically send any queued notifications via a worker in the
    privileged process
  - The API endpoint paths are prefixed with /config/notifications
  - API endpoints which read/modify notification system config require
    Sys.Audit (read) or Sys.Modify (modify/test endpoint) permissions
    on /system/notifications

  - tape-{backup,restore} settings and datastore options now have a
    'notification-mode' parameter, which allows to choose between the
    'legacy' behavior (sendmail to a selected user's email address) and
    the event-based notification system. If the parameter is not set, we
    default to the legacy behavior in order to keep existing behavior
    as is for now. For new datastores/tape backup jobs created
    interactively from the UI, we set the parameter to 'notification-system'
    and opt in into the new system by default. For the CLI/API we don't
    so that we don't cause problems with any automations/scripts -
    there I'd change the default to the new system with the next major release

Rough edges:
  - Datastore option view in UI could be improved. When 'notification-mode' is
    set to 'notification-system', we should indicate that 'mailto-user' and the
    other notification settings have no effect. Already did that in the edit
    window, but in the grid panel I did not find a quick way to do that.

Prerequisites:

  - This patch series requires the patches for 'proxmox' from
  https://lists.proxmox.com/pipermail/pve-devel/2024-April/063493.html
  to be applied first

  - The dependency for proxmox-widget-toolkit should be bumped to at least
  4.1.4 (we need: "utils: add mechanism to add and override translatable notification event
  descriptions in the product specific UIs")

Changes since v4:
  - add __default__ for notification mode combo boxes to avoid issues with
    existing jobs/datastores
  - allow users with Datastore.Backup to see datastores in matcher value suggestions
  - fix copy/paste error in docs
  - cargo fmt

Changes since v3:
  - Pull in UI changes from 'notification metadata matching improvments' as
    component overrides - this can and should be removed once the
    widget toolkit patches are applied.
  - no other changes apart from that

Changes since v2:
  - consolidate verify/prune/sync/tape-job to job-id
  - add a note to docs that package update notifications
    are only sent by nodes with an active subscription

Changes since v1:
  - proxmox_notify: Change default-matcher to exclude 'prune success' notifications
  - proxmox_notify: Move sendmail/forward functions from proxmox_sys
  - proxmox_notify: Improve docs for API types
  - proxmox-widget-toolkit: allow to override default mail author
  - Add documentation, man pages, etc. (mostly copied from PVE, some
    stylistic touchups, some rephrasing)
  - Add additional matchable metadata fields, allowing to match on
    individual jobs:
    - verification-job
    - prune-job
    - sync-job
    - tape-job
  - Sort queued notifications by timestamp before sending
  - Fix missing hostname in package-update notification
  - Change default mail author in the UI from Proxmox VE to Proxmox Backup Server ($hostname)

proxmox:

Lukas Wagner (5):
  notify: expose `config` module
  notify: use std::sync::OnceCell instead of lazy_static!
  notify: pbs-context: exclude successful prunes in default matcher
  notify: endpoints: matcher: improve descriptions for API types
  notify: add getter for notification timestamp

 proxmox-notify/Cargo.toml                |  1 -
 proxmox-notify/src/config.rs             | 23 ++++++++++++-------
 proxmox-notify/src/context/pbs.rs        |  5 ++++-
 proxmox-notify/src/endpoints/gotify.rs   |  6 ++---
 proxmox-notify/src/endpoints/sendmail.rs | 15 ++++++++-----
 proxmox-notify/src/endpoints/smtp.rs     | 28 +++++++++++++++---------
 proxmox-notify/src/lib.rs                | 11 ++++++----
 proxmox-notify/src/matcher.rs            | 18 +++++++--------
 8 files changed, 66 insertions(+), 41 deletions(-)


proxmox-widget-toolkit:

Lukas Wagner (1):
  sendmail: smtp: allow to overide default mail author

 src/Schema.js                  | 9 +++++++++
 src/panel/SendmailEditPanel.js | 2 +-
 src/panel/SmtpEditPanel.js     | 2 +-
 src/window/EndpointEditBase.js | 1 +
 4 files changed, 12 insertions(+), 2 deletions(-)


proxmox-backup:

Lukas Wagner (38):
  pbs-config: add module for loading notification config
  server: rename email_notifications module to notifications
  notifications: allow sending notifications via proxmox_notify
  buildsys: install templates for test notifications
  pbs-config: acl: add /system/notifications as known ACL path
  api: add endpoints for querying/testing notification targets
  api: add endpoints for notification matchers
  api: add endpoints for sendmail targets
  api: add endpoints for smtp targets
  api: add endpoints for gotify targets
  api: add endpoints for querying known notification values/fields
  api-types: api: datatore: add notification-mode parameter
  api-types: api: tape: add notification-mode parameter
  server: notifications: send GC notifications via notification system
  server: notifications: send prune notifications via notification
    system
  server: notifications: send verify notifications via notification
    system
  server: notifications: send sync notifications via notification system
  server: notifications: send update notifications via notification
    system
  server: notifications: send acme notifications via notification system
  server: notifications: send tape notifications via notification system
  ui: add notification config panel
  ui: tape backup job: add selector for notification-mode
  ui: tape backup: add selector for 'notification-mode'
  ui: tape restore: add 'notification-mode' parameter
  ui: datastore options: add 'notification-mode' parameter
  ui: utils: add overrides for known notification metadata fields/values
  ui: datastore edit: make new stores use notification system by default
  ui: permissions paths: add /system/notifications to combobox
  proxmox-backup-manager: add CLI for notification targets
  proxmox-backup-manager: add CLI for notification matchers
  proxmox-backup-manager: add CLI for gotify endpoints
  proxmox-backup-manager: add CLI for sendmail endpoints
  proxmox-backup-manager: add CLI for SMTP endpoints
  docgen: generate synopsis for notifications{-priv,}.cfg
  docs: add documentation for notification system
  ui: util: override default mail author for sendmail/smtp targets
  ui: notifications: pull in UX improvements for match rules creation
  api: notification: also list datastores if user has only Backup privs

 Cargo.toml                                    |    3 +
 Makefile                                      |    5 +-
 debian/proxmox-backup-server.install          |   31 +
 docs/Makefile                                 |    6 +-
 docs/conf.py                                  |    2 +
 docs/config/notifications-priv/format.rst     |    1 +
 docs/config/notifications-priv/man5.rst       |   24 +
 docs/config/notifications/format.rst          |    2 +
 docs/config/notifications/man5.rst            |   24 +
 docs/configuration-files.rst                  |   30 +
 docs/index.rst                                |    1 +
 docs/notifications.rst                        |  213 ++++
 pbs-api-types/src/datastore.rs                |   22 +
 pbs-api-types/src/jobs.rs                     |    8 +-
 pbs-config/Cargo.toml                         |    1 +
 pbs-config/src/acl.rs                         |    3 +-
 pbs-config/src/lib.rs                         |    1 +
 pbs-config/src/notifications.rs               |   41 +
 src/api2/config/datastore.rs                  |    9 +
 src/api2/config/mod.rs                        |    2 +
 src/api2/config/notifications/gotify.rs       |  190 +++
 src/api2/config/notifications/matchers.rs     |  170 +++
 src/api2/config/notifications/mod.rs          |  205 +++
 src/api2/config/notifications/sendmail.rs     |  178 +++
 src/api2/config/notifications/smtp.rs         |  191 +++
 src/api2/config/notifications/targets.rs      |   63 +
 src/api2/config/tape_backup_job.rs            |    8 +
 src/api2/pull.rs                              |   10 +-
 src/api2/tape/backup.rs                       |   62 +-
 src/api2/tape/restore.rs                      |   46 +-
 src/bin/docgen.rs                             |    4 +
 src/bin/proxmox-backup-api.rs                 |   11 +
 src/bin/proxmox-backup-manager.rs             |    2 +
 src/bin/proxmox-backup-proxy.rs               |    1 +
 src/bin/proxmox_backup_manager/mod.rs         |    2 +
 .../notifications/gotify.rs                   |   93 ++
 .../notifications/matchers.rs                 |   93 ++
 .../notifications/mod.rs                      |   21 +
 .../notifications/sendmail.rs                 |   94 ++
 .../notifications/smtp.rs                     |   96 ++
 .../notifications/targets.rs                  |   51 +
 src/server/email_notifications.rs             |  763 ------------
 src/server/gc_job.rs                          |   10 +-
 src/server/mod.rs                             |    4 +-
 src/server/notifications.rs                   |  566 +++++++++
 src/server/verify_job.rs                      |   10 +-
 src/tape/drive/mod.rs                         |   22 +-
 src/tape/mod.rs                               |   27 +
 src/tape/pool_writer/mod.rs                   |   11 +-
 templates/Makefile                            |   41 +
 templates/default/acme-err-body.txt.hbs       |    7 +
 templates/default/acme-err-subject.txt.hbs    |    1 +
 templates/default/gc-err-body.txt.hbs         |    8 +
 templates/default/gc-err-subject.txt.hbs      |    1 +
 templates/default/gc-ok-body.txt.hbs          |   23 +
 templates/default/gc-ok-subject.txt.hbs       |    1 +
 .../default/package-updates-body.txt.hbs      |    8 +
 .../default/package-updates-subject.txt.hbs   |    1 +
 templates/default/prune-err-body.txt.hbs      |   10 +
 templates/default/prune-err-subject.txt.hbs   |    1 +
 templates/default/prune-ok-body.txt.hbs       |   10 +
 templates/default/prune-ok-subject.txt.hbs    |    1 +
 templates/default/sync-err-body.txt.hbs       |   14 +
 templates/default/sync-err-subject.txt.hbs    |    5 +
 templates/default/sync-ok-body.txt.hbs        |   14 +
 templates/default/sync-ok-subject.txt.hbs     |    5 +
 .../default/tape-backup-err-body.txt.hbs      |   26 +
 .../default/tape-backup-err-subject.txt.hbs   |    5 +
 templates/default/tape-backup-ok-body.txt.hbs |   27 +
 .../default/tape-backup-ok-subject.txt.hbs    |    5 +
 templates/default/tape-load-body.txt.hbs      |   15 +
 templates/default/tape-load-subject.txt.hbs   |    1 +
 templates/default/test-body.html.hbs          |    1 +
 templates/default/test-body.txt.hbs           |    1 +
 templates/default/test-subject.txt.hbs        |    1 +
 templates/default/verify-err-body.txt.hbs     |   14 +
 templates/default/verify-err-subject.txt.hbs  |    1 +
 templates/default/verify-ok-body.txt.hbs      |   10 +
 templates/default/verify-ok-subject.txt.hbs   |    1 +
 www/Makefile                                  |    2 +
 www/NavigationTree.js                         |    6 +
 www/OnlineHelpInfo.js                         |   24 +
 www/Utils.js                                  |   39 +
 www/config/NotificationConfigView.js          |   11 +
 www/datastore/OptionView.js                   |   15 +
 www/form/PermissionPathSelector.js            |    1 +
 www/tape/window/TapeBackup.js                 |   25 +
 www/tape/window/TapeBackupJob.js              |   33 +
 www/tape/window/TapeRestore.js                |   17 +
 www/window/DataStoreEdit.js                   |   13 +
 www/window/NotificationMatcherOverride.js     | 1105 +++++++++++++++++
 www/window/NotifyOptions.js                   |   44 +
 92 files changed, 4158 insertions(+), 869 deletions(-)
 create mode 100644 docs/config/notifications-priv/format.rst
 create mode 100644 docs/config/notifications-priv/man5.rst
 create mode 100644 docs/config/notifications/format.rst
 create mode 100644 docs/config/notifications/man5.rst
 create mode 100644 docs/notifications.rst
 create mode 100644 pbs-config/src/notifications.rs
 create mode 100644 src/api2/config/notifications/gotify.rs
 create mode 100644 src/api2/config/notifications/matchers.rs
 create mode 100644 src/api2/config/notifications/mod.rs
 create mode 100644 src/api2/config/notifications/sendmail.rs
 create mode 100644 src/api2/config/notifications/smtp.rs
 create mode 100644 src/api2/config/notifications/targets.rs
 create mode 100644 src/bin/proxmox_backup_manager/notifications/gotify.rs
 create mode 100644 src/bin/proxmox_backup_manager/notifications/matchers.rs
 create mode 100644 src/bin/proxmox_backup_manager/notifications/mod.rs
 create mode 100644 src/bin/proxmox_backup_manager/notifications/sendmail.rs
 create mode 100644 src/bin/proxmox_backup_manager/notifications/smtp.rs
 create mode 100644 src/bin/proxmox_backup_manager/notifications/targets.rs
 delete mode 100644 src/server/email_notifications.rs
 create mode 100644 src/server/notifications.rs
 create mode 100644 templates/Makefile
 create mode 100644 templates/default/acme-err-body.txt.hbs
 create mode 100644 templates/default/acme-err-subject.txt.hbs
 create mode 100644 templates/default/gc-err-body.txt.hbs
 create mode 100644 templates/default/gc-err-subject.txt.hbs
 create mode 100644 templates/default/gc-ok-body.txt.hbs
 create mode 100644 templates/default/gc-ok-subject.txt.hbs
 create mode 100644 templates/default/package-updates-body.txt.hbs
 create mode 100644 templates/default/package-updates-subject.txt.hbs
 create mode 100644 templates/default/prune-err-body.txt.hbs
 create mode 100644 templates/default/prune-err-subject.txt.hbs
 create mode 100644 templates/default/prune-ok-body.txt.hbs
 create mode 100644 templates/default/prune-ok-subject.txt.hbs
 create mode 100644 templates/default/sync-err-body.txt.hbs
 create mode 100644 templates/default/sync-err-subject.txt.hbs
 create mode 100644 templates/default/sync-ok-body.txt.hbs
 create mode 100644 templates/default/sync-ok-subject.txt.hbs
 create mode 100644 templates/default/tape-backup-err-body.txt.hbs
 create mode 100644 templates/default/tape-backup-err-subject.txt.hbs
 create mode 100644 templates/default/tape-backup-ok-body.txt.hbs
 create mode 100644 templates/default/tape-backup-ok-subject.txt.hbs
 create mode 100644 templates/default/tape-load-body.txt.hbs
 create mode 100644 templates/default/tape-load-subject.txt.hbs
 create mode 100644 templates/default/test-body.html.hbs
 create mode 100644 templates/default/test-body.txt.hbs
 create mode 100644 templates/default/test-subject.txt.hbs
 create mode 100644 templates/default/verify-err-body.txt.hbs
 create mode 100644 templates/default/verify-err-subject.txt.hbs
 create mode 100644 templates/default/verify-ok-body.txt.hbs
 create mode 100644 templates/default/verify-ok-subject.txt.hbs
 create mode 100644 www/config/NotificationConfigView.js
 create mode 100644 www/window/NotificationMatcherOverride.js


Summary over all repositories:
  104 files changed, 4236 insertions(+), 912 deletions(-)

-- 
Generated by git-murpp 0.7.1


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel


^ permalink raw reply	[flat|nested] 47+ messages in thread

end of thread, other threads:[~2024-04-23 22:23 UTC | newest]

Thread overview: 47+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-04-23 11:51 [pbs-devel] [PATCH many v5 00/44] integrate notification system Lukas Wagner
2024-04-23 11:51 ` [pbs-devel] [PATCH proxmox v5 01/44] notify: expose `config` module Lukas Wagner
2024-04-23 11:51 ` [pbs-devel] [PATCH proxmox v5 02/44] notify: use std::sync::OnceCell instead of lazy_static! Lukas Wagner
2024-04-23 11:51 ` [pbs-devel] [PATCH proxmox v5 03/44] notify: pbs-context: exclude successful prunes in default matcher Lukas Wagner
2024-04-23 11:51 ` [pbs-devel] [PATCH proxmox v5 04/44] notify: endpoints: matcher: improve descriptions for API types Lukas Wagner
2024-04-23 11:51 ` [pbs-devel] [PATCH proxmox v5 05/44] notify: add getter for notification timestamp Lukas Wagner
2024-04-23 11:51 ` [pbs-devel] [PATCH widget-toolkit v5 06/44] sendmail: smtp: allow to overide default mail author Lukas Wagner
2024-04-23 17:35   ` [pbs-devel] applied: " Thomas Lamprecht
2024-04-23 11:51 ` [pbs-devel] [PATCH proxmox-backup v5 07/44] pbs-config: add module for loading notification config Lukas Wagner
2024-04-23 11:51 ` [pbs-devel] [PATCH proxmox-backup v5 08/44] server: rename email_notifications module to notifications Lukas Wagner
2024-04-23 11:51 ` [pbs-devel] [PATCH proxmox-backup v5 09/44] notifications: allow sending notifications via proxmox_notify Lukas Wagner
2024-04-23 11:51 ` [pbs-devel] [PATCH proxmox-backup v5 10/44] buildsys: install templates for test notifications Lukas Wagner
2024-04-23 11:51 ` [pbs-devel] [PATCH proxmox-backup v5 11/44] pbs-config: acl: add /system/notifications as known ACL path Lukas Wagner
2024-04-23 11:51 ` [pbs-devel] [PATCH proxmox-backup v5 12/44] api: add endpoints for querying/testing notification targets Lukas Wagner
2024-04-23 11:51 ` [pbs-devel] [PATCH proxmox-backup v5 13/44] api: add endpoints for notification matchers Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 14/44] api: add endpoints for sendmail targets Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 15/44] api: add endpoints for smtp targets Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 16/44] api: add endpoints for gotify targets Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 17/44] api: add endpoints for querying known notification values/fields Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 18/44] api-types: api: datatore: add notification-mode parameter Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 19/44] api-types: api: tape: " Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 20/44] server: notifications: send GC notifications via notification system Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 21/44] server: notifications: send prune " Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 22/44] server: notifications: send verify " Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 23/44] server: notifications: send sync " Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 24/44] server: notifications: send update " Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 25/44] server: notifications: send acme " Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 26/44] server: notifications: send tape " Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 27/44] ui: add notification config panel Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 28/44] ui: tape backup job: add selector for notification-mode Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 29/44] ui: tape backup: add selector for 'notification-mode' Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 30/44] ui: tape restore: add 'notification-mode' parameter Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 31/44] ui: datastore options: " Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 32/44] ui: utils: add overrides for known notification metadata fields/values Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 33/44] ui: datastore edit: make new stores use notification system by default Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 34/44] ui: permissions paths: add /system/notifications to combobox Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 35/44] proxmox-backup-manager: add CLI for notification targets Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 36/44] proxmox-backup-manager: add CLI for notification matchers Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 37/44] proxmox-backup-manager: add CLI for gotify endpoints Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 38/44] proxmox-backup-manager: add CLI for sendmail endpoints Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 39/44] proxmox-backup-manager: add CLI for SMTP endpoints Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 40/44] docgen: generate synopsis for notifications{-priv, }.cfg Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 41/44] docs: add documentation for notification system Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 42/44] ui: util: override default mail author for sendmail/smtp targets Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 43/44] ui: notifications: pull in UX improvements for match rules creation Lukas Wagner
2024-04-23 11:52 ` [pbs-devel] [PATCH proxmox-backup v5 44/44] api: notification: also list datastores if user has only Backup privs Lukas Wagner
2024-04-23 22:23 ` [pbs-devel] applied-series: [PATCH many v5 00/44] integrate notification system Thomas Lamprecht

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal