From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 7150B1FF0B7 for ; Fri, 13 Nov 2026 16:14:09 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id CBCAA21AA9; Fri, 13 Nov 2026 16:10:06 +0100 (CET) From: Lukas Wagner To: pve-devel@lists.proxmox.com Subject: [PATCH many 00/55] notify: maintenance, PDM preparations, queue + worker Date: Fri, 13 Nov 2026 16:07:33 +0100 Message-ID: <20261113150828.756836-1-l.wagner@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794582522914 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.205 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_CSS_A 0.1 Contains URL's A record listed in the Spamhaus CSS blocklist [139.162.176.150] Message-ID-Hash: FW3AGYXUCZXP52OSZOPVG3PK2MYCIULQ X-Message-ID-Hash: FW3AGYXUCZXP52OSZOPVG3PK2MYCIULQ X-MailFrom: l.wagner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: The notification stack will soon be integrated into a new product (PDM), so it makes sense to use this opportunity for any pending breaking changes and general cleanup of the notification stack, partly driven by the worker implementation, pending feature work (notification history), and overdue maintenance work to reduce technical debt. The introduction of a shared worker implementation lifts long-standing limitations in PVE, where only the privileged process could send notifications. The worker will also be used in PDM. ## Refactoring - Refactoring the proxmox_notify::Error type Using the old type was quite annoying (one had to use Error::Generic nearly everywhere), the new one applies the lessons-learned from the old one. While this is technically a breaking change, none of the callers used returned errors other than logging/stringifying them, so the callers need no change - Pass context along with calls, wrapped in a context wrapper type, instead of setting it via set_context - Dropped features - gotify - webhook - sendmail - smtp - mail-forwarder The intention was for products to be able to opt into the endpoints they need. Up until now, there was no instance of one endpoint type not being used by a product, hence they are dropped and replaced by a single 'send-impl' feature. If we ever introduce a type of endpoint that will only used by a single product, we can still introduce a new flag for this one. - New feature flags - send-impl: everything needed to *send* a notification - api-impl: API handler implementations - api-types: API types; useful for the Rust-based PDM UI once we integrate it there - queue-submitter: facilities to submit a notification to the notiticiation queue - queue-worker: implementation of the notification worker and queue consumer - restructuring of module paths - lib.rs was huge, so it was split up into appropriate submodules - api types were moved into an 'api_types' submodule - cleanup of the public crate interface - many types didn't need to be public, so these were removed from the public interface to ease further maintenance - switch to typed section config - don't repeatedly re-parse builtin-configuration; do it once and store it a OnceLock - offer builder for instantiating notifications. This allow us to extend notifications with new optional fields later. - Notifications get a `recipient` field, where a recipient can be a the regular matcher system, a target, or a list of email addresses. The 'target' mode allows us to drop the special-cased 'test_target' API. Now, testing a a target is just a regular send, the 'send receipts' feature described later allows us continue reporting any error to the user when testing a target. The 'email' is a shortcut that creates an implicit 'sendmail' endpoint, which allows us to simplify the 'legacy-sendmail' mode in the application code. Now, it uses a regular proxmox_notify::send and does not need to fake a notification config first. The main motivation of this changes were the introduction of the notification queue and planning of the notification history. The 'recipient' approach allows us to unify a lot of code paths, removing special cases that would need to be considered. Now, *any* notification, goes through the queue and will be eventually be persisted in the notification queue. ## Features - send receipts `proxmox_notify::send` did not return any indiciation whether the notification was sent successfully, it was only logged. This series now returns a 'receipt', that includes information about which matcher matched a notification, which targets were notified, including any errors (per target). Eventually, this receipt will also be recorded in the notification history, giving users a clear picture about which notification was sent when, where and why. - notification queue / worker PBS already used a queue-based approach for sending notifications from the unprivileged proxy process. This series adds a generic queue and worker implementation to proxmox_notify which can be used by all products (PVE and soon PDM). This resolves a long standing issue where a hanging notification send call would would block the cluster resource manager for possibly extended periods of time ## User-facing changes - The switch to the queue-based approach means that any log message produced by the notification stack will not be shown in a task log when sending a notification. For now, these will only show up the system logs. In the future, I'd like to integrate this with the notification history, in the history view it should be possible to retrieve the log messages that were logged for any notification (by entering a tracing span with the notification UUID set as a field, which makes it possible to filter the system log for messages for a specific notification) - In PVE, there is a new daemon 'pve-notify-worker' This one is the host for the notification worker task in PVE. ## Bumps - proxmox-notify should receive a major version bump (2 -> 3); the API breakage is deliberate for cleaning up tech debt and preparing for the integration in PDM. - libpve-rs-perl must depend on proxmox-notify v3 - proxmox-backup must depend on proxmox-notify v3 - pve-notify-worker (part of pve-manager) needs a first release - libpve-notify-perl must depend on bumped libpve-rs-perl (enqueue method) - pve-manager must depend on bumped libpve-notify-perl (simplified interface for legacy-sendmail) - proxmox-mail-forward must depend on proxmox-notify v3 - proxmox-mail-forward might need *some* dependency on pve-notify-worker, otherwise notifications might be deferred or dropped if proxmox-mail-forward is updated, but the worker for the queue is not installed or running yet. In PBS, the regular proxmox-backup-server packages houses the worker, so there we'd have a circular dependecy, which is of course also bad. Not sure what the best approach here is, I'm happy for any suggestions. proxmox: Lukas Wagner (42): notify: disallow `unsafe` code notify: refactor error type notify: move `Endpoint` trait to endpoint module as pub(crate) notify: use ConfigDigest instead of raw byte arrays notify: remove static context and pass context as parameter notify: matcher: drop unused field matcher verifiers notify: introduce 'send-impl', 'api-impl' and 'api-types' features notify: matcher: make matcher validation methods public notify: make SendmailEndpoint and SmtpEndpoint private types notify: move `send`, `test_target` and `Bus` to new `send` module notify: move Origin type to api_types module notify: move Target type to api_types module notify: move Config to config module notify: move Notification and sub-types to `notification` module notify: move schema module to api_types notify: move check_matches from api_types::matcher to send::Bus notify: move TemplateSource to context module notify: config: cleanup helpers names and visibility notify: config: rename section config parser/writer methods notify: config: pass builtins explicitly when parsing section config notify: implement PartialEq for all Config types notify: use typed SectionConfigData notify: api: api types: rename Deleteable* -> Deletable* notify: context: parse builtin config only once notify: context: parse proxy config in trait impl, not at the callsite notify: gate sub-modules with the appropriate feature flags notify: fold mail-forwarder feature into send-impl notify: forwarded mails: don't hardcode severity and metadata fields notify: don't pass uid when instantiating forwarded mail notifications notify: derive PartialEq for Notification type notify: add getters for severity and fields on Notification type notify: add test suite for Notification type notify: serialize raw, forwarded mail content as base64 notify: unify code paths for legacy-sendmail and regular notifications notify: add backwards-compat note to Notification type notify: remove 'test_target' API notify: replace notification instantiation with builder pattern notify: add send receipts notify: implement notification queue submitter notify: context: add read_config notify: queue: add queue consumer notify: queue: add worker task Cargo.toml | 1 + proxmox-notify/Cargo.toml | 72 +- proxmox-notify/src/api/common.rs | 42 +- proxmox-notify/src/api/gotify.rs | 122 ++- proxmox-notify/src/api/matcher.rs | 148 ++-- proxmox-notify/src/api/mod.rs | 197 ++--- proxmox-notify/src/api/sendmail.rs | 76 +- proxmox-notify/src/api/smtp.rs | 133 ++- proxmox-notify/src/api/webhook.rs | 176 ++-- proxmox-notify/src/api_types/gotify.rs | 70 ++ .../src/{ => api_types}/matcher/calendar.rs | 25 +- .../src/{ => api_types}/matcher/expression.rs | 2 +- .../src/{ => api_types}/matcher/field.rs | 101 ++- .../src/{ => api_types}/matcher/mod.rs | 100 +-- .../src/{ => api_types}/matcher/severity.rs | 24 +- proxmox-notify/src/api_types/mod.rs | 79 ++ proxmox-notify/src/api_types/receipt.rs | 138 ++++ proxmox-notify/src/{ => api_types}/schema.rs | 0 proxmox-notify/src/api_types/sendmail.rs | 95 +++ proxmox-notify/src/api_types/smtp.rs | 137 ++++ proxmox-notify/src/api_types/webhook.rs | 211 +++++ proxmox-notify/src/config.rs | 427 ++++++---- proxmox-notify/src/context/mod.rs | 79 +- proxmox-notify/src/context/pbs.rs | 52 +- proxmox-notify/src/context/pve.rs | 52 +- proxmox-notify/src/context/test.rs | 50 +- proxmox-notify/src/endpoints/common/mail.rs | 29 +- proxmox-notify/src/endpoints/common/mod.rs | 1 - proxmox-notify/src/endpoints/gotify.rs | 107 +-- proxmox-notify/src/endpoints/mod.rs | 20 +- proxmox-notify/src/endpoints/sendmail.rs | 137 +--- proxmox-notify/src/endpoints/smtp.rs | 209 +---- proxmox-notify/src/endpoints/webhook.rs | 307 ++----- proxmox-notify/src/error.rs | 165 ++++ proxmox-notify/src/lib.rs | 754 +----------------- proxmox-notify/src/notification.rs | 423 ++++++++++ proxmox-notify/src/queue/consumer.rs | 418 ++++++++++ proxmox-notify/src/queue/mod.rs | 29 + proxmox-notify/src/queue/submitter.rs | 59 ++ proxmox-notify/src/queue/worker.rs | 638 +++++++++++++++ proxmox-notify/src/renderer/mod.rs | 44 +- proxmox-notify/src/send.rs | 487 +++++++++++ .../testdata/notifications-priv.cfg.ref | 8 + proxmox-notify/testdata/notifications.cfg.ref | 29 + 44 files changed, 4245 insertions(+), 2228 deletions(-) create mode 100644 proxmox-notify/src/api_types/gotify.rs rename proxmox-notify/src/{ => api_types}/matcher/calendar.rs (82%) rename proxmox-notify/src/{ => api_types}/matcher/expression.rs (98%) rename proxmox-notify/src/{ => api_types}/matcher/field.rs (73%) rename proxmox-notify/src/{ => api_types}/matcher/mod.rs (84%) rename proxmox-notify/src/{ => api_types}/matcher/severity.rs (83%) create mode 100644 proxmox-notify/src/api_types/mod.rs create mode 100644 proxmox-notify/src/api_types/receipt.rs rename proxmox-notify/src/{ => api_types}/schema.rs (100%) create mode 100644 proxmox-notify/src/api_types/sendmail.rs create mode 100644 proxmox-notify/src/api_types/smtp.rs create mode 100644 proxmox-notify/src/api_types/webhook.rs create mode 100644 proxmox-notify/src/error.rs create mode 100644 proxmox-notify/src/notification.rs create mode 100644 proxmox-notify/src/queue/consumer.rs create mode 100644 proxmox-notify/src/queue/mod.rs create mode 100644 proxmox-notify/src/queue/submitter.rs create mode 100644 proxmox-notify/src/queue/worker.rs create mode 100644 proxmox-notify/src/send.rs create mode 100644 proxmox-notify/testdata/notifications-priv.cfg.ref create mode 100644 proxmox-notify/testdata/notifications.cfg.ref proxmox-backup: Lukas Wagner (4): notifications: don't ignore verification errors in legacy-sendmail mode notifications: adapt to proxmox-notify API changes api: notification: send test notification via proxmox_notify::send notifications: send all notifications via proxmox-notify's queue Cargo.toml | 13 +- pbs-config/Cargo.toml | 10 +- pbs-config/src/notifications.rs | 10 +- src/api2/config/notifications/gotify.rs | 31 +- src/api2/config/notifications/matchers.rs | 29 +- src/api2/config/notifications/sendmail.rs | 29 +- src/api2/config/notifications/smtp.rs | 29 +- src/api2/config/notifications/targets.rs | 46 +- src/api2/config/notifications/webhook.rs | 29 +- src/bin/docgen.rs | 7 +- src/bin/proxmox-backup-api.rs | 3 +- src/bin/proxmox-backup-manager.rs | 1 - src/bin/proxmox-backup-proxy.rs | 1 - src/bin/proxmox-daily-update.rs | 3 - .../notifications/gotify.rs | 2 +- .../notifications/matchers.rs | 2 +- .../notifications/sendmail.rs | 2 +- .../notifications/smtp.rs | 2 +- .../notifications/webhook.rs | 2 +- src/server/notifications/mod.rs | 582 ++++++++---------- 20 files changed, 409 insertions(+), 424 deletions(-) proxmox-perl-rs: Lukas Wagner (2): pve: notify: adapt to changes in the public API of proxmox-notify pve: notify: add `PVE::RS::Notify::enqueue` method pve-rs/Cargo.toml | 2 +- pve-rs/src/bindings/mod.rs | 3 - pve-rs/src/bindings/notify.rs | 237 +++++++++++++++++++++++----------- pve-rs/src/lib.rs | 29 +++-- 4 files changed, 179 insertions(+), 92 deletions(-) pve-cluster: Lukas Wagner (1): notify: send notifications via notification queue src/PVE/Notify.pm | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) pve-manager: Lukas Wagner (3): vzdump: send legacy-sendmail notifications via simplified API pve-notify-worker: add new rust-based notification worker daemon debian: add 'pve-notify-worker' as a dependency PVE/VZDump.pm | 28 +- debian/control | 1 + rust/.cargo/config.toml | 5 + rust/.gitignore | 11 + rust/Cargo.toml | 47 ++ rust/LICENSE | 661 ++++++++++++++++++ rust/Makefile | 118 ++++ rust/debian/changelog | 6 + rust/debian/control | 36 + rust/debian/copyright | 19 + rust/debian/docs | 1 + rust/debian/pve-notify-worker.service | 17 + rust/debian/pve-notify-worker.tmpfiles | 3 + rust/debian/rules | 45 ++ .../scripts/elf-strip-unused-dependencies.sh | 20 + rust/debian/source/format | 1 + rust/debian/source/lintian-overrides | 0 rust/defines.mk | 14 + rust/lib/pve-buildcfg/Cargo.toml | 9 + rust/lib/pve-buildcfg/build.rs | 17 + rust/lib/pve-buildcfg/src/lib.rs | 37 + rust/pve-notify-worker/Cargo.toml | 24 + rust/pve-notify-worker/src/main.rs | 69 ++ rust/rustfmt.toml | 2 + 24 files changed, 1165 insertions(+), 26 deletions(-) create mode 100644 rust/.cargo/config.toml create mode 100644 rust/.gitignore create mode 100644 rust/Cargo.toml create mode 100644 rust/LICENSE create mode 100644 rust/Makefile create mode 100644 rust/debian/changelog create mode 100644 rust/debian/control create mode 100644 rust/debian/copyright create mode 100644 rust/debian/docs create mode 100644 rust/debian/pve-notify-worker.service create mode 100644 rust/debian/pve-notify-worker.tmpfiles create mode 100755 rust/debian/rules create mode 100755 rust/debian/scripts/elf-strip-unused-dependencies.sh create mode 100644 rust/debian/source/format create mode 100644 rust/debian/source/lintian-overrides create mode 100644 rust/defines.mk create mode 100644 rust/lib/pve-buildcfg/Cargo.toml create mode 100644 rust/lib/pve-buildcfg/build.rs create mode 100644 rust/lib/pve-buildcfg/src/lib.rs create mode 100644 rust/pve-notify-worker/Cargo.toml create mode 100644 rust/pve-notify-worker/src/main.rs create mode 100644 rust/rustfmt.toml proxmox-mail-forward: Lukas Wagner (3): forward mails via notification queue use return code to signal permanent or temporary failure limit size of mail read from stdin Cargo.toml | 5 +- src/main.rs | 315 +++++++++++++++++++++++++++++++++++----------------- 2 files changed, 217 insertions(+), 103 deletions(-) Summary over all repositories: 95 files changed, 6231 insertions(+), 2875 deletions(-) -- Generated by murpp 0.12.1