From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id A42D599D9C for ; Tue, 14 Nov 2023 14:01:13 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 3600D3018D for ; Tue, 14 Nov 2023 14:00:29 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Tue, 14 Nov 2023 14:00:25 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 0892A429C4 for ; Tue, 14 Nov 2023 14:00:22 +0100 (CET) From: Lukas Wagner To: pve-devel@lists.proxmox.com Date: Tue, 14 Nov 2023 13:59:54 +0100 Message-Id: <20231114130000.565122-47-l.wagner@proxmox.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231114130000.565122-1-l.wagner@proxmox.com> References: <20231114130000.565122-1-l.wagner@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.411 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_ASCII_DIVIDERS 0.8 Email that uses ascii formatting dividers and possible spam tricks KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - WEIRD_PORT 0.001 Uses non-standard port number for HTTP Subject: [pve-devel] [PATCH v2 pve-docs 46/52] notifications: update docs to for matcher-based notifications X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Nov 2023 13:01:13 -0000 Target groups and filters have been replaced by notification matchers. The matcher can match on certain notification properties and route the notification to a target in case of a match. This patch updates the docs to reflect these changes. Signed-off-by: Lukas Wagner --- notifications.adoc | 254 +++++++++++++++++++++++++++++++-------------- 1 file changed, 174 insertions(+), 80 deletions(-) diff --git a/notifications.adoc b/notifications.adoc index c4d2931..764ec72 100644 --- a/notifications.adoc +++ b/notifications.adoc @@ -5,45 +5,40 @@ ifndef::manvolnum[] :pve-toplevel: endif::manvolnum[] -[[notification_events]] -Notification Events -------------------- - -{pve} will attempt to notify system administrators in case of certain events, -such as: - -[width="100%",options="header"] -|=========================================================================== -| Event name | Description | Severity -| `package-updates` | System updates are available | `info` -| `fencing` | The {pve} HA manager has fenced a node | `error` -| `replication` | A storage replication job has failed | `error` -| `vzdump` | vzdump backup finished | `info` (`error` on failure) -|=========================================================================== - -In the 'Notification' panel of the datacenter view, the system's behavior can be -configured for all events except backup jobs. For backup jobs, -the settings can be found in the respective backup job configuration. -For every notification event there is an option to configure the notification -behavior (*when* to send a notification) and the notification target (*where* to -send the notification). - - -See also: - -* xref:datacenter_configuration_file[Datacenter Configuration] -* xref:datacenter_configuration_file[vzdump] +Overview +-------- + +{pve} will send notifications if case of noteworthy events in the system. + +There are a number of different xref:notification_events[notification events], +each with their own set of metadata fields that can be used in +notification matchers. + +A xref:notification_matchers[notification matcher] determines +_which_ notifications shall be sent _where_. +A matcher has _match rules_, that can be used to +match on certain notification properties (e.g. timestamp, severity, +metadata fields). +If a matcher matches a notification, the notification will be routed +to a configurable set of notification targets. + +A xref:notification_targets[notification target] is an abstraction for a +destination where a notification should be sent to - for instance, +a Gotify server instance, or a set of email addresses. +There are multiple types of notification targets, including +`sendmail`, which uses the system's sendmail command to send emails, +or `gotify`, which sends a notification to a Gotify instance. + +The notification system can be configured in the GUI under +Datacenter -> Notifications. The configuration is stored in +`/etc/pve/notifications.cfg` and `/etc/pve/priv/notifications.cfg` - +the latter contains sensitive configuration options such as +passwords or authentication tokens for notification targets. [[notification_targets]] Notification Targets -------------------- -Notification targets can be configured in the 'Notification Targets' panel. - -NOTE: The `mail-to-root` target is always available and cannot be modified or -removed. It sends a mail the `root@pam` user by using the `sendmail` command and -serves as a fallback target if no other target is configured for an event. - Sendmail ~~~~~~~~ The sendmail binary is a program commonly found on Unix-like operating systems @@ -73,7 +68,15 @@ set, the plugin will fall back to the `email_from` setting from `datacenter.cfg`. If that is also not set, the plugin will default to `root@$hostname`, where `$hostname` is the hostname of the node. -* `filter`: The name of the filter to use for this target. +Example configuration (`/etc/pve/notifications.cfg`): +---- +sendmail: example + mailto-user root@pam + mailto-user admin@pve + mailto max@example.com + from-address pve1@example.com + comment Send to multiple users/addresses +---- Gotify ~~~~~~ @@ -88,72 +91,163 @@ The configuration for Gotify target plugins has the following options: * `server`: The base URL of the Gotify server, e.g. `http://:8888` * `token`: The authentication token. Tokens can be generated within the Gotify web interface. -* `filter`: The name of the filter to use for this target. NOTE: The Gotify target plugin will respect the HTTP proxy settings from the xref:datacenter_configuration_file[datacenter configuration] -Group -~~~~~ +Example configuration (`/etc/pve/notifications.cfg`): +---- +gotify: example + server http://gotify.example.com:8888 + comment Send to multiple users/addresses +---- -One can only select a single target for notification events. -To notify via multiple targets at the same time, a group can be created. -A group can reference multiple targets. If a group is used as a target, -the notification will be sent to all referenced targets. Groups can reference -all targets except other groups. +The matching entry in `/etc/pve/priv/notifications.cfg`, containing the +secret token: +---- +gotify: example + token somesecrettoken +---- +[[notification_matchers]] +Notification Matchers +--------------------- +Notification matchers route notifications to notification targets based +on their matching rules. These rules can match of certain properties of +a notification, such as the timestamp (`match-calendar`), the severity of +the notificaiton (`match-severity`) or metadata fiels (`match-field`). +If a matcher matches a notification, all targets configured for the matcher +will receive the notification. + +An arbitrary number of matchers can be created, each with with their own +matching rules and targets to notify. +Every target is notified at most once for every notification, even if +the target is used in multiple matchers. + +A matcher without any matching rules is always true; the configured targets +will always be notified. +---- +matcher: always-matches + target admin + comment This matcher always matches +---- -Notification Filters --------------------- -A notification target can be configured to use a *notification filter*. -If a notification is sent to a target with a filter, the -filter will determine if the notification will be actually sent or not. +Matcher Options +~~~~~~~~~~~~~~~ -The following matchers are available: +* `target`: Determine which target should be notified if the matcher matches. +can be used multiple times to notify multiple targets. +* `invert-match`: Inverts the result of the whole matcher +* `mode`: Determines how the individual match rules are evaluated to compute +the result for the whole matcher. If set to `all`, all matching rules must +match. If set to `any`, at least one rule must match. +a matcher must be true. Defaults to `all`. +* `match-calendar`: Match the notification's timestamp against a schedule +* `match-field`: Match the notification's metadata fields +* `match-severity`: Match the notification's severity -* `min-severity`: Matches notifications with equal or higher severity +Calendar Matching Rules +~~~~~~~~~~~~~~~~~~~~~~~ +A calendar matcher matches the time when a notification is sent agaist a +configurable schedule. -It is also possible to configure the evaluation of the individual matchers: +* `match-calendar 8-12` +* `match-calendar 8:00-15:30` +* `match-calendar mon-fri 9:00-17:00` +* `match-calendar sun,tue-wed,fri 9-17` -* `invert-match`: Inverts the result of the whole filter -* `mode`: Sets the logical operator used to connect the individual matchers to -`and` or `or`. Defaults to `and`. +Field Matching Rules +~~~~~~~~~~~~~~~~~~~~ +Notifications have a selection of metadata fields that can be matched. -The `mode` option also influences the evaluation of filters without any -matchers. If set to `or`, an empty filter evaluates to `false` (do not notify). -If set to `and`, the result is `true` (send a notification). ----- -filter: always-matches - mode and +* `match-field exact:type=vzdump` Only match notifications about backups. +* `match-field regex:hostname=^.+\.example\.com$` Match the hostname of +the node. -filter: never-matches - mode or ----- +If a matched metadata field does not exist, the notification will not be +matched. +For instance, a `match-field regex:hostname=.*` directive will only match +notifications that have an arbitraty `hostname` metadata field, but will +not match if the field does not exist. -Permissions ------------ +Severity Matching Rules +~~~~~~~~~~~~~~~~~~~~~~~ +A notification has a associated severity that can be matched. -For every target or filter, there exists a corresponding ACL path -`/mapping/notification/`. -If an operation can be triggered by a user (e.g. via the GUI or API) and if -that operation is configured to notify via a given target, then -the user must have the `Mapping.Use` permission on the corresponding -node in the ACL tree. -`Mapping.Modify` and `Mapping.Audit` are needed for -writing/reading the configuration of a target or filter. +* `match-severity error`: Only match errors +* `match-severity warning,error`: Match warnings and error -NOTE: For backwards-compatibility, the special `mail-to-root` target -does not require `Mapping.Use`. +The following severities are in use: +`info`, `notice`, `warning`, `error`. -NOTE: When sending notifications via a group target, -the user must have the `Mapping.Use` permission for every single endpoint -included in the group. If a group/endpoint is configured to -use a filter, the user must have the `Mapping.Use` permission for the filter -as well. +Examples +~~~~~~~~ +---- +matcher: workday + match-calendar mon-fri 9-17 + target admin + comment Notify admins during working hours + +matcher: night-and-weekend + match-calendar mon-fri 9-17 + invert-match true + target on-call-admins + comment Separate target for non-working hours +---- +---- +matcher: backup-failures + match-field exact:type=vzdump + match-severity error + target backup-admins + comment Send notifications about backup failures to one group of admins + +matcher: cluster-failures + match-field exact:type=replication + match-field exact:type=fencing + mode any + target cluster-admins + comment Send cluster-related notifications to other group of admins +---- +The last matcher could also be rewritten using a field matcher with a regular +expression: +---- +matcher: cluster-failures + match-field regex:type=^(replication|fencing)$ + target cluster-admins + comment Send cluster-related notifications to other group of admins +---- +[[notification_events]] +Notification Events +------------------- +[width="100%",options="header"] +|=========================================================================== +| Event | `type` | Severity | Metadata fields (in addition to `type`) +| System updates available |`package-updates` | `info` | `hostname` +| Cluster node fenced |`fencing` | `error` | `hostname` +| Storage replication failed |`replication` | `error` | - +| Backup finished |`vzdump` | `info` (`error` on failure) | `hostname` +|=========================================================================== +[width="100%",options="header"] +|======================================================================= +| Field name | Description +| `type` | Type of the notifcation +| `hostname` | Hostname, including domain (e.g. `pve1.example.com`) +|======================================================================= +Permissions +----------- + +For every target, there exists a corresponding ACL path +`/mapping/notification/targets/`. Matchers use +a seperate namespace in the ACL tree: `/mapping/notification/matchers/`. + +To test a target, a user must have the `Mapping.Use` permission on the corresponding +node in the ACL tree. +`Mapping.Modify` and `Mapping.Audit` are needed to read/modify the +configuration of a target or matcher. -- 2.39.2