From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <l.wagner@proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange X25519 server-signature RSA-PSS (2048 bits))
 (No client certificate requested)
 by lists.proxmox.com (Postfix) with ESMTPS id A42D599D9C
 for <pve-devel@lists.proxmox.com>; Tue, 14 Nov 2023 14:01:13 +0100 (CET)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
 by firstgate.proxmox.com (Proxmox) with ESMTP id 3600D3018D
 for <pve-devel@lists.proxmox.com>; Tue, 14 Nov 2023 14:00:29 +0100 (CET)
Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com
 [94.136.29.106])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange X25519 server-signature RSA-PSS (2048 bits))
 (No client certificate requested)
 by firstgate.proxmox.com (Proxmox) with ESMTPS
 for <pve-devel@lists.proxmox.com>; Tue, 14 Nov 2023 14:00:25 +0100 (CET)
Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1])
 by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 0892A429C4
 for <pve-devel@lists.proxmox.com>; Tue, 14 Nov 2023 14:00:22 +0100 (CET)
From: Lukas Wagner <l.wagner@proxmox.com>
To: pve-devel@lists.proxmox.com
Date: Tue, 14 Nov 2023 13:59:54 +0100
Message-Id: <20231114130000.565122-47-l.wagner@proxmox.com>
X-Mailer: git-send-email 2.39.2
In-Reply-To: <20231114130000.565122-1-l.wagner@proxmox.com>
References: <20231114130000.565122-1-l.wagner@proxmox.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-SPAM-LEVEL: Spam detection results:  0
 AWL -0.411 Adjusted score from AWL reputation of From: address
 BAYES_00                 -1.9 Bayes spam probability is 0 to 1%
 DMARC_MISSING             0.1 Missing DMARC policy
 KAM_ASCII_DIVIDERS 0.8 Email that uses ascii formatting dividers and possible
 spam tricks
 KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
 SPF_HELO_NONE           0.001 SPF: HELO does not publish an SPF Record
 SPF_PASS               -0.001 SPF: sender matches SPF record
 T_SCC_BODY_TEXT_LINE    -0.01 -
 WEIRD_PORT              0.001 Uses non-standard port number for HTTP
Subject: [pve-devel] [PATCH v2 pve-docs 46/52] notifications: update docs to
 for matcher-based notifications
X-BeenThere: pve-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>, 
 <mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/>
List-Post: <mailto:pve-devel@lists.proxmox.com>
List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>, 
 <mailto:pve-devel-request@lists.proxmox.com?subject=subscribe>
X-List-Received-Date: Tue, 14 Nov 2023 13:01:13 -0000

Target groups and filters have been replaced by notification matchers.
The matcher can match on certain notification properties and route
the notification to a target in case of a match.

This patch updates the docs to reflect these changes.

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
---
 notifications.adoc | 254 +++++++++++++++++++++++++++++++--------------
 1 file changed, 174 insertions(+), 80 deletions(-)

diff --git a/notifications.adoc b/notifications.adoc
index c4d2931..764ec72 100644
--- a/notifications.adoc
+++ b/notifications.adoc
@@ -5,45 +5,40 @@ ifndef::manvolnum[]
 :pve-toplevel:
 endif::manvolnum[]
 
-[[notification_events]]
-Notification Events
--------------------
-
-{pve} will attempt to notify system administrators in case of certain events,
-such as:
-
-[width="100%",options="header"]
-|===========================================================================
-| Event name        | Description                             | Severity
-| `package-updates` | System updates are available            | `info`
-| `fencing`         | The {pve} HA manager has fenced a node  | `error`
-| `replication`     | A storage replication job has failed    | `error`
-| `vzdump`          | vzdump backup finished                  | `info` (`error` on failure)
-|===========================================================================
-
-In the 'Notification' panel of the datacenter view, the system's behavior can be
-configured for all events except backup jobs. For backup jobs,
-the settings can be found in the respective backup job configuration.
-For every notification event there is an option to configure the notification
-behavior (*when* to send a notification) and the notification target (*where* to
-send the notification).
-
-
-See also:
-
-* xref:datacenter_configuration_file[Datacenter Configuration]
-* xref:datacenter_configuration_file[vzdump]
+Overview
+--------
+
+{pve} will send notifications if case of noteworthy events in the system.
+
+There are a number of different xref:notification_events[notification events],
+each with their own set of metadata fields that can be used in
+notification matchers.
+
+A xref:notification_matchers[notification matcher] determines
+_which_ notifications shall be sent _where_.
+A matcher has _match rules_, that can be used to
+match on certain notification properties (e.g. timestamp, severity,
+metadata fields).
+If a matcher matches a notification, the notification will be routed
+to a configurable set of notification targets.
+
+A xref:notification_targets[notification target] is an abstraction for a
+destination where a notification should be sent to - for instance,
+a Gotify server instance, or a set of email addresses.
+There are multiple types of notification targets, including
+`sendmail`, which uses the system's sendmail command to send emails,
+or `gotify`, which sends a notification to a Gotify instance.
+
+The notification system can be configured in the GUI under
+Datacenter -> Notifications. The configuration is stored in
+`/etc/pve/notifications.cfg` and `/etc/pve/priv/notifications.cfg` -
+the latter contains sensitive configuration options such as
+passwords or authentication tokens for notification targets.
 
 [[notification_targets]]
 Notification Targets
 --------------------
 
-Notification targets can be configured in the 'Notification Targets' panel.
-
-NOTE: The `mail-to-root` target is always available and cannot be modified or
-removed. It sends a mail the `root@pam` user by using the `sendmail` command and
-serves as a fallback target if no other target is configured for an event.
-
 Sendmail
 ~~~~~~~~
 The sendmail binary is a program commonly found on Unix-like operating systems
@@ -73,7 +68,15 @@ set, the plugin will fall back to the `email_from` setting from
 `datacenter.cfg`. If that is also not set, the plugin will default to
 `root@$hostname`, where `$hostname` is the hostname of the node.
 
-* `filter`: The name of the filter to use for this target.
+Example configuration (`/etc/pve/notifications.cfg`):
+----
+sendmail: example
+        mailto-user root@pam
+        mailto-user admin@pve
+        mailto max@example.com
+        from-address pve1@example.com
+        comment Send to multiple users/addresses
+----
 
 Gotify
 ~~~~~~
@@ -88,72 +91,163 @@ The configuration for Gotify target plugins has the following options:
 * `server`: The base URL of the Gotify server, e.g. `http://<ip>:8888`
 * `token`: The authentication token. Tokens can be generated within the Gotify
 web interface.
-* `filter`: The name of the filter to use for this target.
 
 NOTE: The Gotify target plugin will respect the HTTP proxy settings from the
  xref:datacenter_configuration_file[datacenter configuration]
 
-Group
-~~~~~
+Example configuration (`/etc/pve/notifications.cfg`):
+----
+gotify: example
+        server http://gotify.example.com:8888
+        comment Send to multiple users/addresses
+----
 
-One can only select a single target for notification events.
-To notify via multiple targets at the same time, a group can be created.
-A group can reference multiple targets. If a group is used as a target,
-the notification will be sent to all referenced targets. Groups can reference
-all targets except other groups.
+The matching entry in `/etc/pve/priv/notifications.cfg`, containing the
+secret token:
+----
+gotify: example
+        token somesecrettoken
+----
 
+[[notification_matchers]]
+Notification Matchers
+---------------------
+Notification matchers route notifications to notification targets based
+on their matching rules. These rules can match of certain properties of
+a notification, such as the timestamp (`match-calendar`), the severity of
+the notificaiton (`match-severity`) or metadata fiels (`match-field`).
+If a matcher matches a notification, all targets configured for the matcher
+will receive the notification.
+
+An arbitrary number of matchers can be created, each with with their own
+matching rules and targets to notify.
+Every target is notified at most once for every notification, even if
+the target is used in multiple matchers.
+
+A matcher without any matching rules is always true; the configured targets
+will always be notified.
+----
+matcher: always-matches
+        target admin
+        comment This matcher always matches
+----
 
-Notification Filters
---------------------
-A notification target can be configured to use a *notification filter*.
-If a notification is sent to a target with a filter, the
-filter will determine if the notification will be actually sent or not.
+Matcher Options
+~~~~~~~~~~~~~~~
 
-The following matchers are available:
+* `target`: Determine which target should be notified if the matcher matches.
+can be used multiple times to notify multiple targets.
+* `invert-match`: Inverts the result of the whole matcher
+* `mode`: Determines how the individual match rules are evaluated to compute
+the result for the whole matcher. If set to `all`, all matching rules must
+match. If set to `any`, at least one rule must match.
+a matcher must be true. Defaults to `all`.
+* `match-calendar`: Match the notification's timestamp against a schedule
+* `match-field`: Match the notification's metadata fields
+* `match-severity`: Match the notification's severity
 
-* `min-severity`: Matches notifications with equal or higher severity
+Calendar Matching Rules
+~~~~~~~~~~~~~~~~~~~~~~~
+A calendar matcher matches the time when a notification is sent agaist a
+configurable schedule.
 
-It is also possible to configure the evaluation of the individual matchers:
+* `match-calendar 8-12`
+* `match-calendar 8:00-15:30`
+* `match-calendar mon-fri 9:00-17:00`
+* `match-calendar sun,tue-wed,fri 9-17`
 
-* `invert-match`: Inverts the result of the whole filter
-* `mode`: Sets the logical operator used to connect the individual matchers to
-`and` or `or`. Defaults to `and`.
+Field Matching Rules
+~~~~~~~~~~~~~~~~~~~~
+Notifications have a selection of metadata fields that can be matched.
 
-The `mode` option also influences the evaluation of filters without any
-matchers. If set to `or`, an empty filter evaluates to `false` (do not notify).
-If set to `and`, the result is `true` (send a notification).
-----
-filter: always-matches
-    mode and
+* `match-field exact:type=vzdump` Only match notifications about backups.
+* `match-field regex:hostname=^.+\.example\.com$` Match the hostname of
+the node.
 
-filter: never-matches
-    mode or
-----
+If a matched metadata field does not exist, the notification will not be
+matched.
+For instance, a `match-field regex:hostname=.*` directive will only match
+notifications that have an arbitraty `hostname` metadata field, but will
+not match if the field does not exist.
 
-Permissions
------------
+Severity Matching Rules
+~~~~~~~~~~~~~~~~~~~~~~~
+A notification has a associated severity that can be matched.
 
-For every target or filter, there exists a corresponding ACL path
-`/mapping/notification/<name>`.
-If an operation can be triggered by a user (e.g. via the GUI or API) and if
-that operation is configured to notify via a given target, then
-the user must have the `Mapping.Use` permission on the corresponding
-node in the ACL tree.
-`Mapping.Modify` and `Mapping.Audit` are needed for
-writing/reading the configuration of a target or filter.
+* `match-severity error`: Only match errors
+* `match-severity warning,error`: Match warnings and error
 
-NOTE: For backwards-compatibility, the special `mail-to-root` target
-does not require `Mapping.Use`.
+The following severities are in use:
+`info`, `notice`, `warning`, `error`.
 
-NOTE: When sending notifications via a group target,
-the user must have the `Mapping.Use` permission for every single endpoint
-included in the group. If a group/endpoint is configured to
-use a filter, the user must have the `Mapping.Use` permission for the filter
-as well.
 
+Examples
+~~~~~~~~
+----
+matcher: workday
+        match-calendar mon-fri 9-17
+        target admin
+        comment Notify admins during working hours
+
+matcher: night-and-weekend
+        match-calendar mon-fri 9-17
+        invert-match true
+        target on-call-admins
+        comment Separate target for non-working hours
+----
 
+----
+matcher: backup-failures
+        match-field exact:type=vzdump
+        match-severity error
+        target backup-admins
+        comment Send notifications about backup failures to one group of admins
+
+matcher: cluster-failures
+        match-field exact:type=replication
+        match-field exact:type=fencing
+        mode any
+        target cluster-admins
+        comment Send cluster-related notifications to other group of admins
+----
 
+The last matcher could also be rewritten using a field matcher with a regular
+expression:
+----
+matcher: cluster-failures
+        match-field regex:type=^(replication|fencing)$
+        target cluster-admins
+        comment Send cluster-related notifications to other group of admins
+----
 
+[[notification_events]]
+Notification Events
+-------------------
 
+[width="100%",options="header"]
+|===========================================================================
+| Event                        | `type`            | Severity | Metadata fields (in addition to `type`)
+| System updates available     |`package-updates`  | `info`   | `hostname`
+| Cluster node fenced          |`fencing`          | `error`  | `hostname`
+| Storage replication failed   |`replication`      | `error`  | -
+| Backup finished              |`vzdump`           | `info` (`error` on failure) | `hostname`
+|===========================================================================
 
+[width="100%",options="header"]
+|=======================================================================
+| Field name | Description
+| `type`     | Type of the notifcation
+| `hostname` | Hostname, including domain (e.g. `pve1.example.com`)
+|=======================================================================
 
+Permissions
+-----------
+
+For every target, there exists a corresponding ACL path
+`/mapping/notification/targets/<name>`. Matchers use
+a seperate namespace in the ACL tree: `/mapping/notification/matchers/<name>`.
+
+To test a target, a user must have the `Mapping.Use` permission on the corresponding
+node in the ACL tree.
+`Mapping.Modify` and `Mapping.Audit` are needed to read/modify the
+configuration of a target or matcher.
-- 
2.39.2