From: Lukas Wagner <l.wagner@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH many 00/55] notify: maintenance, PDM preparations, queue + worker
Date: Fri, 13 Nov 2026 16:07:33 +0100 [thread overview]
Message-ID: <20261113150828.756836-1-l.wagner@proxmox.com> (raw)
The notification stack will soon be integrated into a new product (PDM), so it
makes sense to use this opportunity for any pending breaking changes and
general cleanup of the notification stack, partly driven by the worker
implementation, pending feature work (notification history), and overdue
maintenance work to reduce technical debt.
The introduction of a shared worker implementation lifts long-standing limitations in PVE,
where only the privileged process could send notifications. The worker will also be used
in PDM.
## Refactoring
- Refactoring the proxmox_notify::Error type
Using the old type was quite annoying (one had to use Error::Generic nearly everywhere), the
new one applies the lessons-learned from the old one.
While this is technically a breaking change, none of the callers used returned errors
other than logging/stringifying them, so the callers need no change
- Pass context along with calls, wrapped in a context wrapper type, instead of
setting it via set_context
- Dropped features
- gotify
- webhook
- sendmail
- smtp
- mail-forwarder
The intention was for products to be able to opt into the endpoints they need.
Up until now, there was no instance of one endpoint type not being used
by a product, hence they are dropped and replaced by a single 'send-impl'
feature. If we ever introduce a type of endpoint that will only used
by a single product, we can still introduce a new flag for this one.
- New feature flags
- send-impl: everything needed to *send* a notification
- api-impl: API handler implementations
- api-types: API types; useful for the Rust-based PDM UI once we integrate it there
- queue-submitter: facilities to submit a notification to the notiticiation queue
- queue-worker: implementation of the notification worker and queue consumer
- restructuring of module paths
- lib.rs was huge, so it was split up into appropriate submodules
- api types were moved into an 'api_types' submodule
- cleanup of the public crate interface
- many types didn't need to be public, so these were removed from the public
interface to ease further maintenance
- switch to typed section config
- don't repeatedly re-parse builtin-configuration; do it once
and store it a OnceLock
- offer builder for instantiating notifications. This allow us
to extend notifications with new optional fields later.
- Notifications get a `recipient` field, where a recipient can be a the regular matcher
system, a target, or a list of email addresses.
The 'target' mode allows us to drop the special-cased 'test_target' API. Now, testing a
a target is just a regular send, the 'send receipts' feature described later allows us
continue reporting any error to the user when testing a target.
The 'email' is a shortcut that creates an implicit 'sendmail' endpoint, which allows
us to simplify the 'legacy-sendmail' mode in the application code. Now, it uses a
regular proxmox_notify::send and does not need to fake a notification config first.
The main motivation of this changes were the introduction of the notification queue and
planning of the notification history. The 'recipient' approach allows us to unify
a lot of code paths, removing special cases that would need to be considered.
Now, *any* notification, goes through the queue and will be eventually be
persisted in the notification queue.
## Features
- send receipts `proxmox_notify::send` did not return any indiciation whether
the notification was sent successfully, it was only logged. This series now
returns a 'receipt', that includes information about which matcher matched a
notification, which targets were notified, including any errors (per target).
Eventually, this receipt will also be recorded in the notification history,
giving users a clear picture about which notification was sent when, where and
why.
- notification queue / worker PBS already used a queue-based approach for
sending notifications from the unprivileged proxy process. This series adds a
generic queue and worker implementation to proxmox_notify which can be used by
all products (PVE and soon PDM). This resolves a long standing issue where a
hanging notification send call would would block the cluster resource manager
for possibly extended periods of time
## User-facing changes
- The switch to the queue-based approach means that any log message produced
by the notification stack will not be shown in a task log when sending a
notification. For now, these will only show up the system logs. In the future,
I'd like to integrate this with the notification history, in the history view
it should be possible to retrieve the log messages that were logged for any
notification (by entering a tracing span with the notification UUID set as a
field, which makes it possible to filter the system log for messages for a
specific notification)
- In PVE, there is a new daemon 'pve-notify-worker'
This one is the host for the notification worker task in PVE.
## Bumps
- proxmox-notify should receive a major version bump (2 -> 3); the API breakage is deliberate
for cleaning up tech debt and preparing for the integration in PDM.
- libpve-rs-perl must depend on proxmox-notify v3
- proxmox-backup must depend on proxmox-notify v3
- pve-notify-worker (part of pve-manager) needs a first release
- libpve-notify-perl must depend on bumped libpve-rs-perl
(enqueue method)
- pve-manager must depend on bumped libpve-notify-perl
(simplified interface for legacy-sendmail)
- proxmox-mail-forward must depend on proxmox-notify v3
- proxmox-mail-forward might need *some* dependency on pve-notify-worker,
otherwise notifications might be deferred or dropped if proxmox-mail-forward is
updated, but the worker for the queue is not installed or running yet. In PBS,
the regular proxmox-backup-server packages houses the worker, so there we'd
have a circular dependecy, which is of course also bad.
Not sure what the best approach here is, I'm happy for any suggestions.
proxmox:
Lukas Wagner (42):
notify: disallow `unsafe` code
notify: refactor error type
notify: move `Endpoint` trait to endpoint module as pub(crate)
notify: use ConfigDigest instead of raw byte arrays
notify: remove static context and pass context as parameter
notify: matcher: drop unused field matcher verifiers
notify: introduce 'send-impl', 'api-impl' and 'api-types' features
notify: matcher: make matcher validation methods public
notify: make SendmailEndpoint and SmtpEndpoint private types
notify: move `send`, `test_target` and `Bus` to new `send` module
notify: move Origin type to api_types module
notify: move Target type to api_types module
notify: move Config to config module
notify: move Notification and sub-types to `notification` module
notify: move schema module to api_types
notify: move check_matches from api_types::matcher to send::Bus
notify: move TemplateSource to context module
notify: config: cleanup helpers names and visibility
notify: config: rename section config parser/writer methods
notify: config: pass builtins explicitly when parsing section config
notify: implement PartialEq for all Config types
notify: use typed SectionConfigData
notify: api: api types: rename Deleteable* -> Deletable*
notify: context: parse builtin config only once
notify: context: parse proxy config in trait impl, not at the callsite
notify: gate sub-modules with the appropriate feature flags
notify: fold mail-forwarder feature into send-impl
notify: forwarded mails: don't hardcode severity and metadata fields
notify: don't pass uid when instantiating forwarded mail notifications
notify: derive PartialEq for Notification type
notify: add getters for severity and fields on Notification type
notify: add test suite for Notification type
notify: serialize raw, forwarded mail content as base64
notify: unify code paths for legacy-sendmail and regular notifications
notify: add backwards-compat note to Notification type
notify: remove 'test_target' API
notify: replace notification instantiation with builder pattern
notify: add send receipts
notify: implement notification queue submitter
notify: context: add read_config
notify: queue: add queue consumer
notify: queue: add worker task
Cargo.toml | 1 +
proxmox-notify/Cargo.toml | 72 +-
proxmox-notify/src/api/common.rs | 42 +-
proxmox-notify/src/api/gotify.rs | 122 ++-
proxmox-notify/src/api/matcher.rs | 148 ++--
proxmox-notify/src/api/mod.rs | 197 ++---
proxmox-notify/src/api/sendmail.rs | 76 +-
proxmox-notify/src/api/smtp.rs | 133 ++-
proxmox-notify/src/api/webhook.rs | 176 ++--
proxmox-notify/src/api_types/gotify.rs | 70 ++
.../src/{ => api_types}/matcher/calendar.rs | 25 +-
.../src/{ => api_types}/matcher/expression.rs | 2 +-
.../src/{ => api_types}/matcher/field.rs | 101 ++-
.../src/{ => api_types}/matcher/mod.rs | 100 +--
.../src/{ => api_types}/matcher/severity.rs | 24 +-
proxmox-notify/src/api_types/mod.rs | 79 ++
proxmox-notify/src/api_types/receipt.rs | 138 ++++
proxmox-notify/src/{ => api_types}/schema.rs | 0
proxmox-notify/src/api_types/sendmail.rs | 95 +++
proxmox-notify/src/api_types/smtp.rs | 137 ++++
proxmox-notify/src/api_types/webhook.rs | 211 +++++
proxmox-notify/src/config.rs | 427 ++++++----
proxmox-notify/src/context/mod.rs | 79 +-
proxmox-notify/src/context/pbs.rs | 52 +-
proxmox-notify/src/context/pve.rs | 52 +-
proxmox-notify/src/context/test.rs | 50 +-
proxmox-notify/src/endpoints/common/mail.rs | 29 +-
proxmox-notify/src/endpoints/common/mod.rs | 1 -
proxmox-notify/src/endpoints/gotify.rs | 107 +--
proxmox-notify/src/endpoints/mod.rs | 20 +-
proxmox-notify/src/endpoints/sendmail.rs | 137 +---
proxmox-notify/src/endpoints/smtp.rs | 209 +----
proxmox-notify/src/endpoints/webhook.rs | 307 ++-----
proxmox-notify/src/error.rs | 165 ++++
proxmox-notify/src/lib.rs | 754 +-----------------
proxmox-notify/src/notification.rs | 423 ++++++++++
proxmox-notify/src/queue/consumer.rs | 418 ++++++++++
proxmox-notify/src/queue/mod.rs | 29 +
proxmox-notify/src/queue/submitter.rs | 59 ++
proxmox-notify/src/queue/worker.rs | 638 +++++++++++++++
proxmox-notify/src/renderer/mod.rs | 44 +-
proxmox-notify/src/send.rs | 487 +++++++++++
.../testdata/notifications-priv.cfg.ref | 8 +
proxmox-notify/testdata/notifications.cfg.ref | 29 +
44 files changed, 4245 insertions(+), 2228 deletions(-)
create mode 100644 proxmox-notify/src/api_types/gotify.rs
rename proxmox-notify/src/{ => api_types}/matcher/calendar.rs (82%)
rename proxmox-notify/src/{ => api_types}/matcher/expression.rs (98%)
rename proxmox-notify/src/{ => api_types}/matcher/field.rs (73%)
rename proxmox-notify/src/{ => api_types}/matcher/mod.rs (84%)
rename proxmox-notify/src/{ => api_types}/matcher/severity.rs (83%)
create mode 100644 proxmox-notify/src/api_types/mod.rs
create mode 100644 proxmox-notify/src/api_types/receipt.rs
rename proxmox-notify/src/{ => api_types}/schema.rs (100%)
create mode 100644 proxmox-notify/src/api_types/sendmail.rs
create mode 100644 proxmox-notify/src/api_types/smtp.rs
create mode 100644 proxmox-notify/src/api_types/webhook.rs
create mode 100644 proxmox-notify/src/error.rs
create mode 100644 proxmox-notify/src/notification.rs
create mode 100644 proxmox-notify/src/queue/consumer.rs
create mode 100644 proxmox-notify/src/queue/mod.rs
create mode 100644 proxmox-notify/src/queue/submitter.rs
create mode 100644 proxmox-notify/src/queue/worker.rs
create mode 100644 proxmox-notify/src/send.rs
create mode 100644 proxmox-notify/testdata/notifications-priv.cfg.ref
create mode 100644 proxmox-notify/testdata/notifications.cfg.ref
proxmox-backup:
Lukas Wagner (4):
notifications: don't ignore verification errors in legacy-sendmail
mode
notifications: adapt to proxmox-notify API changes
api: notification: send test notification via proxmox_notify::send
notifications: send all notifications via proxmox-notify's queue
Cargo.toml | 13 +-
pbs-config/Cargo.toml | 10 +-
pbs-config/src/notifications.rs | 10 +-
src/api2/config/notifications/gotify.rs | 31 +-
src/api2/config/notifications/matchers.rs | 29 +-
src/api2/config/notifications/sendmail.rs | 29 +-
src/api2/config/notifications/smtp.rs | 29 +-
src/api2/config/notifications/targets.rs | 46 +-
src/api2/config/notifications/webhook.rs | 29 +-
src/bin/docgen.rs | 7 +-
src/bin/proxmox-backup-api.rs | 3 +-
src/bin/proxmox-backup-manager.rs | 1 -
src/bin/proxmox-backup-proxy.rs | 1 -
src/bin/proxmox-daily-update.rs | 3 -
.../notifications/gotify.rs | 2 +-
.../notifications/matchers.rs | 2 +-
.../notifications/sendmail.rs | 2 +-
.../notifications/smtp.rs | 2 +-
.../notifications/webhook.rs | 2 +-
src/server/notifications/mod.rs | 582 ++++++++----------
20 files changed, 409 insertions(+), 424 deletions(-)
proxmox-perl-rs:
Lukas Wagner (2):
pve: notify: adapt to changes in the public API of proxmox-notify
pve: notify: add `PVE::RS::Notify::enqueue` method
pve-rs/Cargo.toml | 2 +-
pve-rs/src/bindings/mod.rs | 3 -
pve-rs/src/bindings/notify.rs | 237 +++++++++++++++++++++++-----------
pve-rs/src/lib.rs | 29 +++--
4 files changed, 179 insertions(+), 92 deletions(-)
pve-cluster:
Lukas Wagner (1):
notify: send notifications via notification queue
src/PVE/Notify.pm | 18 ++++++++++++++++--
1 file changed, 16 insertions(+), 2 deletions(-)
pve-manager:
Lukas Wagner (3):
vzdump: send legacy-sendmail notifications via simplified API
pve-notify-worker: add new rust-based notification worker daemon
debian: add 'pve-notify-worker' as a dependency
PVE/VZDump.pm | 28 +-
debian/control | 1 +
rust/.cargo/config.toml | 5 +
rust/.gitignore | 11 +
rust/Cargo.toml | 47 ++
rust/LICENSE | 661 ++++++++++++++++++
rust/Makefile | 118 ++++
rust/debian/changelog | 6 +
rust/debian/control | 36 +
rust/debian/copyright | 19 +
rust/debian/docs | 1 +
rust/debian/pve-notify-worker.service | 17 +
rust/debian/pve-notify-worker.tmpfiles | 3 +
rust/debian/rules | 45 ++
.../scripts/elf-strip-unused-dependencies.sh | 20 +
rust/debian/source/format | 1 +
rust/debian/source/lintian-overrides | 0
rust/defines.mk | 14 +
rust/lib/pve-buildcfg/Cargo.toml | 9 +
rust/lib/pve-buildcfg/build.rs | 17 +
rust/lib/pve-buildcfg/src/lib.rs | 37 +
rust/pve-notify-worker/Cargo.toml | 24 +
rust/pve-notify-worker/src/main.rs | 69 ++
rust/rustfmt.toml | 2 +
24 files changed, 1165 insertions(+), 26 deletions(-)
create mode 100644 rust/.cargo/config.toml
create mode 100644 rust/.gitignore
create mode 100644 rust/Cargo.toml
create mode 100644 rust/LICENSE
create mode 100644 rust/Makefile
create mode 100644 rust/debian/changelog
create mode 100644 rust/debian/control
create mode 100644 rust/debian/copyright
create mode 100644 rust/debian/docs
create mode 100644 rust/debian/pve-notify-worker.service
create mode 100644 rust/debian/pve-notify-worker.tmpfiles
create mode 100755 rust/debian/rules
create mode 100755 rust/debian/scripts/elf-strip-unused-dependencies.sh
create mode 100644 rust/debian/source/format
create mode 100644 rust/debian/source/lintian-overrides
create mode 100644 rust/defines.mk
create mode 100644 rust/lib/pve-buildcfg/Cargo.toml
create mode 100644 rust/lib/pve-buildcfg/build.rs
create mode 100644 rust/lib/pve-buildcfg/src/lib.rs
create mode 100644 rust/pve-notify-worker/Cargo.toml
create mode 100644 rust/pve-notify-worker/src/main.rs
create mode 100644 rust/rustfmt.toml
proxmox-mail-forward:
Lukas Wagner (3):
forward mails via notification queue
use return code to signal permanent or temporary failure
limit size of mail read from stdin
Cargo.toml | 5 +-
src/main.rs | 315 +++++++++++++++++++++++++++++++++++-----------------
2 files changed, 217 insertions(+), 103 deletions(-)
Summary over all repositories:
95 files changed, 6231 insertions(+), 2875 deletions(-)
--
Generated by murpp 0.12.1
next reply other threads:[~2026-11-13 15:14 UTC|newest]
Thread overview: 58+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-13 15:07 Lukas Wagner [this message]
2026-11-13 15:07 ` [PATCH proxmox 01/55] notify: disallow `unsafe` code Lukas Wagner
2026-11-13 15:07 ` [PATCH proxmox 02/55] notify: refactor error type Lukas Wagner
2026-11-13 15:07 ` [PATCH proxmox 03/55] notify: move `Endpoint` trait to endpoint module as pub(crate) Lukas Wagner
2026-11-13 15:07 ` [PATCH proxmox 04/55] notify: use ConfigDigest instead of raw byte arrays Lukas Wagner
2026-11-13 15:07 ` [PATCH proxmox 05/55] notify: remove static context and pass context as parameter Lukas Wagner
2026-11-13 15:07 ` [PATCH proxmox 06/55] notify: matcher: drop unused field matcher verifiers Lukas Wagner
2026-11-13 15:07 ` [PATCH proxmox 07/55] notify: introduce 'send-impl', 'api-impl' and 'api-types' features Lukas Wagner
2026-11-13 15:07 ` [PATCH proxmox 08/55] notify: matcher: make matcher validation methods public Lukas Wagner
2026-11-13 15:07 ` [PATCH proxmox 09/55] notify: make SendmailEndpoint and SmtpEndpoint private types Lukas Wagner
2026-11-13 15:07 ` [PATCH proxmox 10/55] notify: move `send`, `test_target` and `Bus` to new `send` module Lukas Wagner
2026-11-13 15:07 ` [PATCH proxmox 11/55] notify: move Origin type to api_types module Lukas Wagner
2026-11-13 15:07 ` [PATCH proxmox 12/55] notify: move Target " Lukas Wagner
2026-11-13 15:07 ` [PATCH proxmox 13/55] notify: move Config to config module Lukas Wagner
2026-11-13 15:07 ` [PATCH proxmox 14/55] notify: move Notification and sub-types to `notification` module Lukas Wagner
2026-11-13 15:07 ` [PATCH proxmox 15/55] notify: move schema module to api_types Lukas Wagner
2026-11-13 15:07 ` [PATCH proxmox 16/55] notify: move check_matches from api_types::matcher to send::Bus Lukas Wagner
2026-11-13 15:07 ` [PATCH proxmox 17/55] notify: move TemplateSource to context module Lukas Wagner
2026-11-13 15:07 ` [PATCH proxmox 18/55] notify: config: cleanup helpers names and visibility Lukas Wagner
2026-11-13 15:07 ` [PATCH proxmox 19/55] notify: config: rename section config parser/writer methods Lukas Wagner
2026-11-13 15:07 ` [PATCH proxmox 20/55] notify: config: pass builtins explicitly when parsing section config Lukas Wagner
2026-11-13 15:07 ` [PATCH proxmox 21/55] notify: implement PartialEq for all Config types Lukas Wagner
2026-11-13 15:07 ` [PATCH proxmox 22/55] notify: use typed SectionConfigData Lukas Wagner
2026-11-13 15:07 ` [PATCH proxmox 23/55] notify: api: api types: rename Deleteable* -> Deletable* Lukas Wagner
2026-11-13 15:07 ` [PATCH proxmox 24/55] notify: context: parse builtin config only once Lukas Wagner
2026-11-13 15:07 ` [PATCH proxmox 25/55] notify: context: parse proxy config in trait impl, not at the callsite Lukas Wagner
2026-11-13 15:07 ` [PATCH proxmox 26/55] notify: gate sub-modules with the appropriate feature flags Lukas Wagner
2026-11-13 15:08 ` [PATCH proxmox 27/55] notify: fold mail-forwarder feature into send-impl Lukas Wagner
2026-11-13 15:08 ` [PATCH proxmox 28/55] notify: forwarded mails: don't hardcode severity and metadata fields Lukas Wagner
2026-11-13 15:08 ` [PATCH proxmox 29/55] notify: don't pass uid when instantiating forwarded mail notifications Lukas Wagner
2026-11-13 15:08 ` [PATCH proxmox 30/55] notify: derive PartialEq for Notification type Lukas Wagner
2026-11-13 15:08 ` [PATCH proxmox 31/55] notify: add getters for severity and fields on " Lukas Wagner
2026-11-13 15:08 ` [PATCH proxmox 32/55] notify: add test suite for " Lukas Wagner
2026-11-13 15:08 ` [PATCH proxmox 33/55] notify: serialize raw, forwarded mail content as base64 Lukas Wagner
2026-11-13 15:08 ` [PATCH proxmox 34/55] notify: unify code paths for legacy-sendmail and regular notifications Lukas Wagner
2026-11-13 15:08 ` [PATCH proxmox 35/55] notify: add backwards-compat note to Notification type Lukas Wagner
2026-11-13 15:08 ` [PATCH proxmox 36/55] notify: remove 'test_target' API Lukas Wagner
2026-11-13 15:08 ` [PATCH proxmox 37/55] notify: replace notification instantiation with builder pattern Lukas Wagner
2026-11-13 15:08 ` [PATCH proxmox 38/55] notify: add send receipts Lukas Wagner
2026-11-13 15:08 ` [PATCH proxmox 39/55] notify: implement notification queue submitter Lukas Wagner
2026-11-13 15:08 ` [PATCH proxmox 40/55] notify: context: add read_config Lukas Wagner
2026-11-13 15:08 ` [PATCH proxmox 41/55] notify: queue: add queue consumer Lukas Wagner
2026-11-13 15:08 ` [PATCH proxmox 42/55] notify: queue: add worker task Lukas Wagner
2026-11-13 15:08 ` [PATCH proxmox-backup 43/55] notifications: don't ignore verification errors in legacy-sendmail mode Lukas Wagner
2026-11-13 15:08 ` [PATCH proxmox-backup 44/55] notifications: adapt to proxmox-notify API changes Lukas Wagner
2026-11-13 15:08 ` [PATCH proxmox-backup 45/55] api: notification: send test notification via proxmox_notify::send Lukas Wagner
2026-11-13 15:08 ` [PATCH proxmox-backup 46/55] notifications: send all notifications via proxmox-notify's queue Lukas Wagner
2026-11-13 15:08 ` [PATCH proxmox-perl-rs 47/55] pve: notify: adapt to changes in the public API of proxmox-notify Lukas Wagner
2026-11-13 15:08 ` [PATCH proxmox-perl-rs 48/55] pve: notify: add `PVE::RS::Notify::enqueue` method Lukas Wagner
2026-11-13 15:08 ` [PATCH pve-cluster 49/55] notify: send notifications via notification queue Lukas Wagner
2026-11-13 15:08 ` [PATCH manager 50/55] vzdump: send legacy-sendmail notifications via simplified API Lukas Wagner
2026-11-13 15:08 ` [PATCH manager 51/55] pve-notify-worker: add new rust-based notification worker daemon Lukas Wagner
2026-11-13 15:08 ` [PATCH manager 52/55] debian: add 'pve-notify-worker' as a dependency Lukas Wagner
2026-11-13 15:08 ` [PATCH proxmox-mail-forward 53/55] forward mails via notification queue Lukas Wagner
2026-11-13 15:08 ` [PATCH proxmox-mail-forward 54/55] use return code to signal permanent or temporary failure Lukas Wagner
2026-11-13 15:08 ` [PATCH proxmox-mail-forward 55/55] limit size of mail read from stdin Lukas Wagner
2026-11-17 9:35 ` [PATCH many 00/55] notify: maintenance, PDM preparations, queue + worker Lukas Wagner
2026-11-18 9:55 ` superseded: " Lukas Wagner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261113150828.756836-1-l.wagner@proxmox.com \
--to=l.wagner@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.