From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id CB8659556F for ; Fri, 12 Apr 2024 12:07:16 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 3C0267A54 for ; Fri, 12 Apr 2024 12:06:46 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Fri, 12 Apr 2024 12:06:42 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id C8A3F45175 for ; Fri, 12 Apr 2024 12:06:39 +0200 (CEST) From: Lukas Wagner To: pbs-devel@lists.proxmox.com Date: Fri, 12 Apr 2024 12:05:58 +0200 Message-Id: <20240412100631.94218-1-l.wagner@proxmox.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.156 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pbs-devel] [PATCH proxmox-backup 00/33] integrate notification system X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Apr 2024 10:07:16 -0000 These patches integrate the notification system which was introduced in Proxmox VE in 8.1 into Proxmox Backup Server. Some highlights/noteworthy details from the series: - notification template files are installed in /usr/share/proxmox-backup/templates/default and are rendered when a notification is sent - since sending notifications needs to be done from a privileged context (to read protected passwords/tokens from the priv config file), we queue notifications to be sent in /var/lib/proxmox-backup/notifications and periodically send any queued notifications via a worker in the privileged process - The API endpoint paths are prefixed with /config/notifications - API endpoints which read/modify notification system config require Sys.Audit (read) or Sys.Modify (modify) permissions on /system/notifications - tape-{backup,restore} settings and datastore options now have a 'notification-mode' parameter, which allows to choose between the 'legacy' behavior (sendmail to a selected user's email address) and the event-based notification system. If the parameter is not set, we default to the legacy behavior in order to keep existing behavior as is for now. For new datastores/tape backup jobs created interactively from teh UI, we set the parameter to 'notification-system' and opt in into the new system by default. Still missing/rough edges: - documentation (will follow soon) - Datastore option view in UI could be improved. When 'notification-mode' is set to 'notification-system', we should indicate that 'mailto-user' and the other notification settings have no effect. Already did that in the edit window, but in the grid panel I did not find a quick way to do that. - still needs a bit more testing (e.g. ACME notifications) - would greatly appreciate people helping here! Open questions: - Right now the 'default-matcher' matches every notification, same as in PVE. Up until now, datastore prune notifications were only sent on errors, so maybe we should also have the default-matcher work the same? The default matcher would then probably be: - !All - severity: error - match-field: exact:type=prune This patch series requires the patches for 'proxmox' from https://lists.proxmox.com/pipermail/pve-devel/2024-April/062708.html to be applied. The dependency for proxmox-widget-toolkit should be bumped to at least 4.1.4 (we need: "utils: add mechanism to add and override translatable notification event descriptions in the product specific UIs") Also, while not strictly needed, the patches for widget-toolkit from https://lists.proxmox.com/pipermail/pve-devel/2024-February/061992.html make creating matchers much easier - the required API endpoints were already implemented by this series. proxmox-backup: Lukas Wagner (33): pbs-config: add module for loading notification config server: rename email_notifications module to notifications notifications: allow sending notifications via proxmox_notify buildsys: install templates for test notifications pbs-config: acl: add /system/notifications as known ACL path api: add endpoints for querying/testing notification targets api: add endpoints for notification matchers api: add endpoints for sendmail targets api: add endpoints for smtp targets api: add endpoints for gotify targets api: add endpoints for querying known notification values/fields api-types: api: datatore: add notification-mode parameter api-types: api: tape: add notification-mode parameter server: notifications: send GC notifications via notification system server: notifications: send prune notifications via notification system server: notifications: send verify notifications via notification system server: notifications: send sync notifications via notification system server: notifications: send update notifications via notification system server: notifications: send acme notifications via notification system server: notifications: send tape notifications via notification system ui: add notification config panel ui: tape backup job: add selector for notification-mode ui: tape backup: add selector for 'notification-mode' ui: tape restore: add 'notification-mode' parameter ui: datastore options: add 'notification-mode' parameter ui: utils: add overrides for known notification metadata fields/values ui: datastore edit: make new stores use notification system by default ui: permissions paths: add /system/notifications to combobox proxmox-backup-manager: add CLI for notification targets proxmox-backup-manager: add CLI for notification matchers proxmox-backup-manager: add CLI for gotify endpoints proxmox-backup-manager: add CLI for sendmail endpoints proxmox-backup-manager: add CLI for SMTP endpoints Cargo.toml | 3 + Makefile | 5 +- debian/proxmox-backup-server.install | 29 + pbs-api-types/src/datastore.rs | 22 + pbs-api-types/src/jobs.rs | 8 +- pbs-config/Cargo.toml | 1 + pbs-config/src/acl.rs | 3 +- pbs-config/src/lib.rs | 1 + pbs-config/src/notifications.rs | 41 + src/api2/config/datastore.rs | 9 + src/api2/config/mod.rs | 2 + src/api2/config/notifications/gotify.rs | 190 +++++ src/api2/config/notifications/matchers.rs | 170 ++++ src/api2/config/notifications/mod.rs | 175 ++++ src/api2/config/notifications/sendmail.rs | 178 ++++ src/api2/config/notifications/smtp.rs | 191 +++++ src/api2/config/notifications/targets.rs | 63 ++ src/api2/config/tape_backup_job.rs | 8 + src/api2/pull.rs | 10 +- src/api2/tape/backup.rs | 62 +- src/api2/tape/restore.rs | 46 +- src/bin/proxmox-backup-api.rs | 11 + src/bin/proxmox-backup-manager.rs | 2 + src/bin/proxmox-backup-proxy.rs | 1 + src/bin/proxmox_backup_manager/mod.rs | 2 + .../notifications/gotify.rs | 93 +++ .../notifications/matchers.rs | 93 +++ .../notifications/mod.rs | 21 + .../notifications/sendmail.rs | 94 +++ .../notifications/smtp.rs | 96 +++ .../notifications/targets.rs | 51 ++ src/server/email_notifications.rs | 763 ------------------ src/server/gc_job.rs | 10 +- src/server/mod.rs | 4 +- src/server/notifications.rs | 544 +++++++++++++ src/server/verify_job.rs | 10 +- src/tape/drive/mod.rs | 22 +- src/tape/mod.rs | 27 + src/tape/pool_writer/mod.rs | 11 +- templates/Makefile | 41 + templates/default/acme-err-body.txt.hbs | 7 + templates/default/acme-err-subject.txt.hbs | 1 + templates/default/gc-err-body.txt.hbs | 8 + templates/default/gc-err-subject.txt.hbs | 1 + templates/default/gc-ok-body.txt.hbs | 23 + templates/default/gc-ok-subject.txt.hbs | 1 + .../default/package-updates-body.txt.hbs | 8 + .../default/package-updates-subject.txt.hbs | 1 + templates/default/prune-err-body.txt.hbs | 10 + templates/default/prune-err-subject.txt.hbs | 1 + templates/default/prune-ok-body.txt.hbs | 10 + templates/default/prune-ok-subject.txt.hbs | 1 + templates/default/sync-err-body.txt.hbs | 14 + templates/default/sync-err-subject.txt.hbs | 5 + templates/default/sync-ok-body.txt.hbs | 14 + templates/default/sync-ok-subject.txt.hbs | 5 + .../default/tape-backup-err-body.txt.hbs | 26 + .../default/tape-backup-err-subject.txt.hbs | 5 + templates/default/tape-backup-ok-body.txt.hbs | 27 + .../default/tape-backup-ok-subject.txt.hbs | 5 + templates/default/tape-load-body.txt.hbs | 15 + templates/default/tape-load-subject.txt.hbs | 1 + templates/default/test-body.html.hbs | 1 + templates/default/test-body.txt.hbs | 1 + templates/default/test-subject.txt.hbs | 1 + templates/default/verify-err-body.txt.hbs | 14 + templates/default/verify-err-subject.txt.hbs | 1 + templates/default/verify-ok-body.txt.hbs | 10 + templates/default/verify-ok-subject.txt.hbs | 1 + www/Makefile | 1 + www/NavigationTree.js | 6 + www/Utils.js | 16 + www/config/NotificationConfigView.js | 11 + www/datastore/OptionView.js | 15 + www/form/PermissionPathSelector.js | 1 + www/tape/window/TapeBackup.js | 25 + www/tape/window/TapeBackupJob.js | 24 + www/tape/window/TapeRestore.js | 17 + www/window/DataStoreEdit.js | 13 + www/window/NotifyOptions.js | 38 + 80 files changed, 2630 insertions(+), 868 deletions(-) create mode 100644 pbs-config/src/notifications.rs create mode 100644 src/api2/config/notifications/gotify.rs create mode 100644 src/api2/config/notifications/matchers.rs create mode 100644 src/api2/config/notifications/mod.rs create mode 100644 src/api2/config/notifications/sendmail.rs create mode 100644 src/api2/config/notifications/smtp.rs create mode 100644 src/api2/config/notifications/targets.rs create mode 100644 src/bin/proxmox_backup_manager/notifications/gotify.rs create mode 100644 src/bin/proxmox_backup_manager/notifications/matchers.rs create mode 100644 src/bin/proxmox_backup_manager/notifications/mod.rs create mode 100644 src/bin/proxmox_backup_manager/notifications/sendmail.rs create mode 100644 src/bin/proxmox_backup_manager/notifications/smtp.rs create mode 100644 src/bin/proxmox_backup_manager/notifications/targets.rs delete mode 100644 src/server/email_notifications.rs create mode 100644 src/server/notifications.rs create mode 100644 templates/Makefile create mode 100644 templates/default/acme-err-body.txt.hbs create mode 100644 templates/default/acme-err-subject.txt.hbs create mode 100644 templates/default/gc-err-body.txt.hbs create mode 100644 templates/default/gc-err-subject.txt.hbs create mode 100644 templates/default/gc-ok-body.txt.hbs create mode 100644 templates/default/gc-ok-subject.txt.hbs create mode 100644 templates/default/package-updates-body.txt.hbs create mode 100644 templates/default/package-updates-subject.txt.hbs create mode 100644 templates/default/prune-err-body.txt.hbs create mode 100644 templates/default/prune-err-subject.txt.hbs create mode 100644 templates/default/prune-ok-body.txt.hbs create mode 100644 templates/default/prune-ok-subject.txt.hbs create mode 100644 templates/default/sync-err-body.txt.hbs create mode 100644 templates/default/sync-err-subject.txt.hbs create mode 100644 templates/default/sync-ok-body.txt.hbs create mode 100644 templates/default/sync-ok-subject.txt.hbs create mode 100644 templates/default/tape-backup-err-body.txt.hbs create mode 100644 templates/default/tape-backup-err-subject.txt.hbs create mode 100644 templates/default/tape-backup-ok-body.txt.hbs create mode 100644 templates/default/tape-backup-ok-subject.txt.hbs create mode 100644 templates/default/tape-load-body.txt.hbs create mode 100644 templates/default/tape-load-subject.txt.hbs create mode 100644 templates/default/test-body.html.hbs create mode 100644 templates/default/test-body.txt.hbs create mode 100644 templates/default/test-subject.txt.hbs create mode 100644 templates/default/verify-err-body.txt.hbs create mode 100644 templates/default/verify-err-subject.txt.hbs create mode 100644 templates/default/verify-ok-body.txt.hbs create mode 100644 templates/default/verify-ok-subject.txt.hbs create mode 100644 www/config/NotificationConfigView.js Summary over all repositories: 80 files changed, 2630 insertions(+), 868 deletions(-) -- Generated by git-murpp 0.7.1