public inbox for pbs-devel@lists.proxmox.com
 help / color / mirror / Atom feed
* [PATCH many v3 00/39] notifications: add nested match expressions
@ 2026-10-09 12:56 Lukas Wagner
  2026-10-09 12:56 ` [PATCH proxmox v3 01/39] add new proxmox-match-expression crate Lukas Wagner
                   ` (38 more replies)
  0 siblings, 39 replies; 40+ messages in thread
From: Lukas Wagner @ 2026-10-09 12:56 UTC (permalink / raw)
  To: pbs-devel, pve-devel

NOTE: The last patch for pve-manager needs to be updated depending on
which version of pve-manager includes these series! (see patch note)

NOTE: I have a couple of other patches planned for the notification stack (**);
so I guess it would make sense wait a bit before applying this -- just to avoid
too many version bumps and too much maintainer churn. I have a patch series
already in the works that builds on top of this one, so I would prefer to
perform any non-trivial changes to this feature as follow-ups; rebasing the new
series I'm working on right now is a huge pain, as it moves a lot of code
around in proxmox-notify (preparations for proxmox-notify v2, essentially)

This resolves a long-standing limitation of notification matchers. Up until
now, notification match rules could only be one level deep, so one top-level
combinator (all/any) and an arbitrary number of match rules as direct children.
This severely limits which kind of matching behaviors can be represented.

The notification stack was always designed with the possibility of arbitrarily
nested match rules, however, due to limitations in our API and config formats
(lacking support for nested data structures), this was not implemented yet.

A prior attempt to resolve the limitations imposed by the lack of nesting was
to add 'sub-matchers', where one matcher could evaluate the result of another
matcher. After some consideration, I decided not to pursue this approach further,
due to concerns about the UX of that approach [1].

This new series chooses a different approach. Nested match rules can now be
represented by a single matcher in the config. The lack of support for nested
data structures in the API was circumvented by storing a serialized form of the
'rule tree' as a JSON blob in a single new configuration parameter
'expression'. If this new key is set, the old 'match-*', 'invert-match' and
'mode' keys are ignored; 'expression' is intended as a replacement for these
keys. I have not yet marked the old keys as deprecated. Theoretically these can
stay supported for now, however when editing a matcher via the UI, the old keys
will always be translated into an equivalent 'expression'. For any new products
receiving the notification stack, we can drop the support for the old keys by
compiling proxmox-notify without the 'legacy-matchers' flag.

The expression evaluation logic was implemented in a new generic crate
'proxmox-match-expression'. 'proxmox-notify' is the first user of this crate,
however it is generic enough to be potentially useful in other contexts as
well.

Documentation will be added in a later patch series, I will include all
relevant changes to the notification stack in one go.

Bumps:
  - pve-rs needs proxmox-notify bumped
  - pve-cluster (libpve-notify-perl) needs pve-rs and pve-common bumped
  - pve-manager needs pve-cluster (libpve-notify-perl, libpve-cluster-perl),
    pve-rs and widget-toolkit bumped
  - proxmox-backup needs widget-toolkit and proxmox-notify bumped
  - proxmox-notify needs a first release of proxmox-match-expression
  
  - proxmox-mail-forward needs to be rebuilt against bumped proxmox-notify (unless
    it has already been changed to not call into proxmox-notify anymore, see below)

[1] https://lore.proxmox.com/pbs-devel/20250521142309.264719-1-l.wagner@proxmox.com/


(**): Some of the things I want to improve/implement:
   - Migrate PVE to use a worker-based approach, same as in PBS
     -> once that is done, we can fully migrate proxmox-mail-forward
        to use the worker/spool dir approach, instead of using proxmox-notify
        directly
   - Introduce a notification history
   - Refactor error types in proxmox-notify a bit

Changes since v1:
  - Move Proxmox::RS::Notify module to PVE::RS::Notify, since it is only used
    by PVE
  - Fix legacy-sendmail mode, the calling code was not adapted to the new API
    of Proxmox::RS::Notify in v1
  - proxmox-match-expression: all-of([]) now evaluates to true, similar to
    Iterator::all or all([]) in python
  - proxmox-notify: Don't allow empty all-of/any-of/one-of rules (in both UI and API,
    proxmox-match-expression still allow them)
  - Also don't allow empty severity/field matchers (empty arrays of matched values)
  - Fix slight inconsistency with the default notification matcher for PBS;
    now prune notifications with severity 'notice' are included once more
  - Other minor review comments
  - Overall polish of the new matcher UI
    - Add hint about nodes being drag&drop'able
    - Add error field if a node is invalid
    - Fixed erroreous caldendar string generated by the UI under
      certain circumenstances

Changes since v2:
  - re-serialize matcher expression on create/update, ensuring
    that the JSON is minified and stripped of control characters (such
    as line breaks)
  - reject creating/updating matchers with expressions if
    not all nodes in a PVE cluster support it
  - Properly handle nested 'not' nodes in the UI
  - In the new calendar edit view, support entering the time
    in formats such as O:00, 7, 730. The UI will automatically
    normalize the input to the appropriate 24h format (e.g. 07:30)
  - Cherry-picked two patches (patch 31, 32) from Fiona [1] for the cluster
    version check.

  Left untentionally unfixed: Due to the new representation of the default
matcher, a previously serialized but unmodified default matcher might be shown
as 'modified' in the UI after the update. This is merely a cosmetic issue that
can be rectified by pressing "Reset".

Thanks go to Wolfgang and Arthur for their feedback of v1.
Thanks also go to Michael Köppl for his feedback on v2.

[1] https://lore.proxmox.com/pve-devel/20260918160841.128088-1-f.ebner@proxmox.com/


proxmox:

Lukas Wagner (18):
  add new proxmox-match-expression crate
  notify: log which matchers matched on a notification
  notify: promote matcher to dir-style module
  notify: fix doc comment
  notify: matcher: break out severity matcher into submodule
  notify: matcher: break out field matcher into submodule
  notify: matcher: break out calendar matcher into submodule
  notify: matcher: calendar: add basic unit test
  notify: matcher: add InlineSeverityMatcher
  notify: matcher: add InlineFieldMatcher
  notify: matcher: add InlineCalendarMatcher
  notify: matcher: add expression support
  notify: api: support new expression parameter
  notify: api: add `get_matcher_as_expression`
  notify: migrate PBS's and PVE's default matcher to expression syntax
  notify: move legacy matcher keys behind feature flag
  notify: matcher: don't allow empty field/severity matchers
  notify: matcher: re-serialize expression on create/update

 Cargo.toml                                    |   3 +
 proxmox-match-expression/Cargo.toml           |  18 +
 proxmox-match-expression/debian/changelog     |   6 +
 proxmox-match-expression/debian/control       |  34 ++
 proxmox-match-expression/debian/copyright     |  18 +
 proxmox-match-expression/debian/debcargo.toml |   7 +
 proxmox-match-expression/src/lib.rs           | 524 +++++++++++++++++
 proxmox-notify/Cargo.toml                     |   7 +-
 proxmox-notify/src/api/matcher.rs             | 253 ++++++++-
 proxmox-notify/src/context/mod.rs             |   4 +-
 proxmox-notify/src/context/pbs.rs             |  87 ++-
 proxmox-notify/src/context/pve.rs             |  67 ++-
 proxmox-notify/src/context/test.rs            |   9 +-
 proxmox-notify/src/lib.rs                     |  29 +-
 proxmox-notify/src/matcher.rs                 | 532 -----------------
 proxmox-notify/src/matcher/calendar.rs        | 158 ++++++
 proxmox-notify/src/matcher/expression.rs      |  97 ++++
 proxmox-notify/src/matcher/field.rs           | 249 ++++++++
 proxmox-notify/src/matcher/mod.rs             | 537 ++++++++++++++++++
 proxmox-notify/src/matcher/severity.rs        | 119 ++++
 20 files changed, 2159 insertions(+), 599 deletions(-)
 create mode 100644 proxmox-match-expression/Cargo.toml
 create mode 100644 proxmox-match-expression/debian/changelog
 create mode 100644 proxmox-match-expression/debian/control
 create mode 100644 proxmox-match-expression/debian/copyright
 create mode 100644 proxmox-match-expression/debian/debcargo.toml
 create mode 100644 proxmox-match-expression/src/lib.rs
 delete mode 100644 proxmox-notify/src/matcher.rs
 create mode 100644 proxmox-notify/src/matcher/calendar.rs
 create mode 100644 proxmox-notify/src/matcher/expression.rs
 create mode 100644 proxmox-notify/src/matcher/field.rs
 create mode 100644 proxmox-notify/src/matcher/mod.rs
 create mode 100644 proxmox-notify/src/matcher/severity.rs


proxmox-widget-toolkit:

Lukas Wagner (4):
  notifications: increase matcher window width
  notifications: matcher: add support for match expressions
  notifications: matcher: add better calendar editor
  notifications: matcher: consistently use title case for UI elements

 src/Makefile                                  |   1 +
 src/Schema.js                                 |  10 +
 src/Utils.js                                  |   2 +-
 src/css/ext6-pmx.css                          |  51 +
 .../NotificationMatchExpressionEditPanel.js   | 892 ++++++++++++++++++
 src/proxmox-dark/scss/extjs/_treepanel.scss   |   5 +
 src/proxmox-dark/scss/proxmox/_general.scss   |   4 +
 src/window/NotificationMatcherEdit.js         | 358 ++++++-
 8 files changed, 1303 insertions(+), 20 deletions(-)
 create mode 100644 src/panel/NotificationMatchExpressionEditPanel.js


proxmox-backup:

Lukas Wagner (3):
  notification: opt into 'legacy-matchers' feature in proxmox-notify
  api: notification: add 'migrate-to-expression' parameter to
    get_matcher
  ui: notification: enable new matcher UI

 Cargo.toml                                |  2 +-
 src/api2/config/notifications/matchers.rs | 18 ++++++++++++++++--
 www/Utils.js                              |  3 +++
 3 files changed, 20 insertions(+), 3 deletions(-)


proxmox-perl-rs:

Lukas Wagner (4):
  notify: move proxmox-notify bindings from common to pve-rs
  notify: matcher: pass matcher config / updater directly
  notify: opt into 'legacy-matchers' feature in proxmox-notify
  notify: add 'migrate_to_expression' parameter for get_matcher

 common/pkg/Makefile                       |   1 -
 common/src/bindings/mod.rs                |   3 -
 pmg-rs/Cargo.toml                         |   1 -
 pve-rs/Cargo.toml                         |   2 +-
 pve-rs/Makefile                           |   1 +
 pve-rs/src/bindings/mod.rs                |   3 +
 {common => pve-rs}/src/bindings/notify.rs | 212 ++++------------------
 7 files changed, 40 insertions(+), 183 deletions(-)
 rename {common => pve-rs}/src/bindings/notify.rs (73%)


pve-cluster:

Fiona Ebner (1):
  cluster: move pvecfg node version helpers from qemu-server

Lukas Wagner (1):
  notify: adapt to changed module prefix for PVE::RS::Notify

 src/PVE/Cluster.pm | 30 ++++++++++++++++++++++++++++++
 src/PVE/Notify.pm  |  4 ++--
 2 files changed, 32 insertions(+), 2 deletions(-)


pve-common:

Fiona Ebner (1):
  tools: move version_cmp() helper from qemu-server

 src/PVE/Tools.pm | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)


pve-manager:

Lukas Wagner (6):
  vzdump: adapt to changed module prefix for PVE::RS::Notify
  api: notification: pass config/updater directly to rust bindings
  api: notification: get_matcher: add 'migrate-to-expression' parameter
  api: notification: add 'expression' to matcher parameter schema
  ui: notification: enable new matcher UI
  api: notification: reject matchers with expression if not supported by
    all nodes

 PVE/API2/Cluster/Notifications.pm | 195 +++++++++++-------------------
 PVE/VZDump.pm                     |  21 ++--
 www/manager6/Utils.js             |   2 +
 3 files changed, 84 insertions(+), 134 deletions(-)


proxmox-mail-forward:

Lukas Wagner (1):
  cargo: opt into legacy-matchers feature

 Cargo.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


Summary over all repositories:
  45 files changed, 3665 insertions(+), 942 deletions(-)

-- 
Generated by murpp 0.12.1




^ permalink raw reply	[flat|nested] 40+ messages in thread

end of thread, other threads:[~2026-10-09 13:05 UTC | newest]

Thread overview: 40+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-10-09 12:56 [PATCH many v3 00/39] notifications: add nested match expressions Lukas Wagner
2026-10-09 12:56 ` [PATCH proxmox v3 01/39] add new proxmox-match-expression crate Lukas Wagner
2026-10-09 12:56 ` [PATCH proxmox v3 02/39] notify: log which matchers matched on a notification Lukas Wagner
2026-10-09 12:56 ` [PATCH proxmox v3 03/39] notify: promote matcher to dir-style module Lukas Wagner
2026-10-09 12:56 ` [PATCH proxmox v3 04/39] notify: fix doc comment Lukas Wagner
2026-10-09 12:56 ` [PATCH proxmox v3 05/39] notify: matcher: break out severity matcher into submodule Lukas Wagner
2026-10-09 12:56 ` [PATCH proxmox v3 06/39] notify: matcher: break out field " Lukas Wagner
2026-10-09 12:56 ` [PATCH proxmox v3 07/39] notify: matcher: break out calendar " Lukas Wagner
2026-10-09 12:56 ` [PATCH proxmox v3 08/39] notify: matcher: calendar: add basic unit test Lukas Wagner
2026-10-09 12:56 ` [PATCH proxmox v3 09/39] notify: matcher: add InlineSeverityMatcher Lukas Wagner
2026-10-09 12:56 ` [PATCH proxmox v3 10/39] notify: matcher: add InlineFieldMatcher Lukas Wagner
2026-10-09 12:56 ` [PATCH proxmox v3 11/39] notify: matcher: add InlineCalendarMatcher Lukas Wagner
2026-10-09 12:56 ` [PATCH proxmox v3 12/39] notify: matcher: add expression support Lukas Wagner
2026-10-09 12:56 ` [PATCH proxmox v3 13/39] notify: api: support new expression parameter Lukas Wagner
2026-10-09 12:56 ` [PATCH proxmox v3 14/39] notify: api: add `get_matcher_as_expression` Lukas Wagner
2026-10-09 12:56 ` [PATCH proxmox v3 15/39] notify: migrate PBS's and PVE's default matcher to expression syntax Lukas Wagner
2026-10-09 12:56 ` [PATCH proxmox v3 16/39] notify: move legacy matcher keys behind feature flag Lukas Wagner
2026-10-09 12:56 ` [PATCH proxmox v3 17/39] notify: matcher: don't allow empty field/severity matchers Lukas Wagner
2026-10-09 12:56 ` [PATCH proxmox v3 18/39] notify: matcher: re-serialize expression on create/update Lukas Wagner
2026-10-09 12:56 ` [PATCH proxmox-widget-toolkit v3 19/39] notifications: increase matcher window width Lukas Wagner
2026-10-09 12:56 ` [PATCH proxmox-widget-toolkit v3 20/39] notifications: matcher: add support for match expressions Lukas Wagner
2026-10-09 12:56 ` [PATCH proxmox-widget-toolkit v3 21/39] notifications: matcher: add better calendar editor Lukas Wagner
2026-10-09 12:56 ` [PATCH proxmox-widget-toolkit v3 22/39] notifications: matcher: consistently use title case for UI elements Lukas Wagner
2026-10-09 12:56 ` [PATCH proxmox-backup v3 23/39] notification: opt into 'legacy-matchers' feature in proxmox-notify Lukas Wagner
2026-10-09 12:56 ` [PATCH proxmox-backup v3 24/39] api: notification: add 'migrate-to-expression' parameter to get_matcher Lukas Wagner
2026-10-09 12:56 ` [PATCH proxmox-backup v3 25/39] ui: notification: enable new matcher UI Lukas Wagner
2026-10-09 12:56 ` [PATCH proxmox-perl-rs v3 26/39] notify: move proxmox-notify bindings from common to pve-rs Lukas Wagner
2026-10-09 12:56 ` [PATCH proxmox-perl-rs v3 27/39] notify: matcher: pass matcher config / updater directly Lukas Wagner
2026-10-09 12:56 ` [PATCH proxmox-perl-rs v3 28/39] notify: opt into 'legacy-matchers' feature in proxmox-notify Lukas Wagner
2026-10-09 12:56 ` [PATCH proxmox-perl-rs v3 29/39] notify: add 'migrate_to_expression' parameter for get_matcher Lukas Wagner
2026-10-09 12:56 ` [PATCH pve-cluster v3 30/39] notify: adapt to changed module prefix for PVE::RS::Notify Lukas Wagner
2026-10-09 12:56 ` [PATCH pve-cluster v3 31/39] cluster: move pvecfg node version helpers from qemu-server Lukas Wagner
2026-10-09 12:56 ` [PATCH common v3 32/39] tools: move version_cmp() helper " Lukas Wagner
2026-10-09 12:56 ` [PATCH manager v3 33/39] vzdump: adapt to changed module prefix for PVE::RS::Notify Lukas Wagner
2026-10-09 12:56 ` [PATCH manager v3 34/39] api: notification: pass config/updater directly to rust bindings Lukas Wagner
2026-10-09 12:56 ` [PATCH manager v3 35/39] api: notification: get_matcher: add 'migrate-to-expression' parameter Lukas Wagner
2026-10-09 12:56 ` [PATCH manager v3 36/39] api: notification: add 'expression' to matcher parameter schema Lukas Wagner
2026-10-09 12:56 ` [PATCH manager v3 37/39] ui: notification: enable new matcher UI Lukas Wagner
2026-10-09 12:56 ` [PATCH manager v3 38/39] api: notification: reject matchers with expression if not supported by all nodes Lukas Wagner
2026-10-09 12:56 ` [PATCH proxmox-mail-forward v3 39/39] cargo: opt into legacy-matchers feature Lukas Wagner

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal