all lists on lists.proxmox.com
 help / color / mirror / Atom feed
From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com, Lukas Wagner <l.wagner@proxmox.com>
Subject: Re: [PATCH many v2 00/63] notify: maintenance, PDM preparations, queue + worker
Date: Wed, 18 Nov 2026 22:44:45 +0100	[thread overview]
Message-ID: <179503802208.3534257.8475790039140179721.b4-ty@b4> (raw)
In-Reply-To: <20261118094348.273646-1-l.wagner@proxmox.com>

On Wed, 18 Nov 2026 10:42:44 +0100, Lukas Wagner wrote:
> The notification stack will soon be integrated into a new product (PDM), so it
> makes sense to use this opportunity for any pending breaking changes and
> general cleanup of the notification stack, partly driven by the worker
> implementation, pending feature work (notification history), and overdue
> maintenance work to reduce technical debt.
> 
> The introduction of a shared worker implementation lifts long-standing limitations in PVE,
> where only the privileged process could send notifications. The worker will also be used
> in PDM.
> 
> [...]

Applied with some changes on top, thanks!

Folded in, besides typos and docs:
- worker: notifications failing validation, symlinks and other entries
  that cannot be opened are marked broken instead of being requeued
  forever; other read errors after a claim requeue instead of using up
  an attempt.
- moved patch 29 "notify: forwarded mails: don't hardcode severity and
  metadata fields" before patch 28 "notify: fold mail-forwarder feature
  into send-impl", so the default feature build works at each commit.
- mail-forward: worker unit ordered after pve-cluster,
  build-dependencies, and the worker package now ships the queue dir
  tmpfiles entry. dpkg configures it before pve-manager, so the worker
  started without its queue; patch 53 "debian: create notification queue
  directory via tmpfiles" got dropped. The worker is uploaded to the PVE
  repo only.
- PBS: patch 47 "api: notification: send test notification via
  proxmox_notify::send" squashed into patch 46 "notifications: adapt to
  proxmox-notify API changes", so the latter builds on its own; the
  multi-line inline tables in Cargo.toml are single-line again, as they
  need TOML 1.1.

Separate commits:
- notify: no config read while pmxcfs is not mounted; queue files are
  0640 and owned by the submitter, and the queue dir is expected to be
  3770, so submitters cannot change or remove each other's notifications
  (the worker checks the setup at startup); warning if the queue stays
  locked for five minutes; IDs queued twice are left alone; receipts
  record 'mailto' for direct email recipients instead of a
  'legacy-sendmail' target name.
- PBS: the queue dir is fixed up through an O_NOFOLLOW handle instead of
  refusing to start; old spool files are only renamed, not read as root.
- pve-cluster: libpve-notify-perl depends on pve-notify-worker.

Left oper open points, input welcome:
- retrying deliveries that reached no target. I had a variant reusing the
  attempt counter, but it seems to fit better with per-target state
  in the notification history, as then one can ensure nothing gets
  submitted multiple times for each specific target.
- the PVE queue path is defined in pve-rs, mail-forward and the worker;
  moving PVEContext and the path into a PVE crate would give one source.
- the PBS queue sits below the backup-owned state dir, so the backup
  user can still replace it.
- patch 56 "services: request pve-notify-worker.service via Wants" has
  no rationale; what does the extra Wants= guard against?
- mail-forward treats stdin I/O errors like an oversized mail and
  bounces it.
- a pve-docs counterpart for the PBS docs change would be nice

[manager] applied to pve-manager.git (master)
[52/63] vzdump: send legacy-sendmail notifications via simplified API
        https://git.proxmox.com/?p=pve-manager.git;a=commitdiff;h=ba8dd37eb7012ad837dba1831da23e964ee71d63
[53/63] debian: create notification queue directory via tmpfiles
        (no commit info)
[54/63] debian: add 'pve-notify-worker' as dependecy
        https://git.proxmox.com/?p=pve-manager.git;a=commitdiff;h=fe4f131c084a0e077a55678ca1929b1d236db709
[55/63] api: services: add 'pve-notify-worker' as known service
        https://git.proxmox.com/?p=pve-manager.git;a=commitdiff;h=7f87d76b293ecd20ba983e43ac41cd941eb9d9fd
[56/63] services: request pve-notify-worker.service via Wants
        https://git.proxmox.com/?p=pve-manager.git;a=commitdiff;h=618c236fb407d6204980197f8eb0d9d31d2dab5e

[proxmox] applied to git/proxmox.git (master)
[01/63] notify: disallow `unsafe` code
        commit: 8c715db3a42b1ef996784dfbcfde326a5d9a0116
[02/63] notify: refactor error type
        commit: c00b7d2b001af30ed00ab78e6646cb4dab26f104
[03/63] notify: move `Endpoint` trait to endpoint module as pub(crate)
        commit: 089c815ba6d7ce2c4ac516d2f73c5b3905a71260
[04/63] notify: use ConfigDigest instead of raw byte arrays
        commit: f37cfe258c2e741f4936d437c233ff2b95e3ab7f
[05/63] notify: remove static context and pass context as parameter
        commit: 3066b4436db4198445785b733cd3e77c9944381b
[06/63] notify: matcher: drop unused field matcher verifiers
        commit: a9c8845e2b4eb307ec528087909df2098541ed68
[07/63] notify: introduce 'send-impl', 'api-impl' and 'api-types' features
        commit: c0d14243fe83f18c8dadf22638daf304ea5a2a16
[08/63] notify: matcher: make matcher validation methods public
        commit: ed684af62fa34af8e49a10086349e62550a0f258
[09/63] notify: matcher: stub CalendarMatcher::matches on wasm32
        commit: a7ea27438b0d1246db109663c45549a926f16051
[10/63] notify: make SendmailEndpoint and SmtpEndpoint private types
        commit: cafaa7ad378188ee03ef3e8c4fadd1c31867dbe5
[11/63] notify: move `send`, `test_target` and `Bus` to new `send` module
        commit: 172e40456be2b21534458c4ce262746bb915b45d
[12/63] notify: move Origin type to api_types module
        commit: 6664b447f484c6608d536874deceff9d4a4e903b
[13/63] notify: move Target type to api_types module
        commit: 2dcca96b0f2e19518f1baa8b84d680a28b638d22
[14/63] notify: move Config to config module
        commit: 39cda59d301b16b6d5cb2fac3cf1fcd3bb925256
[15/63] notify: move Notification and sub-types to `notification` module
        commit: 66edfc9b337737f7add3ea1c24de43319586d816
[16/63] notify: move schema module to api_types
        commit: b9837a3f02c1055dacd5e0b1d7e9922cc7a026be
[17/63] notify: move check_matches from api_types::matcher to send::Bus
        commit: d6973a53b8737978b88bf2f6fb5f6a6fd3aeb0e3
[18/63] notify: move TemplateSource to context module
        commit: bc95b07be0bde600bb14defa75a7142820da3537
[19/63] notify: config: cleanup helper names and visibility
        commit: ef6e079d1d45896ed05ec37b51dac5aadf1a7eb3
[20/63] notify: config: rename section config parser/writer methods
        commit: bd240427b97f3a00e4feda3e1c6f1fbe15c6e322
[21/63] notify: config: pass builtins explicitly when parsing section config
        commit: 13c6f06ca0ae481c900f8c11c4260386201e0182
[22/63] notify: implement PartialEq for all Config types
        commit: b59510fda0ce184d784d805004ebd8e84a49f15f
[23/63] notify: use typed SectionConfigData
        commit: 79b95f7c2b5236080571bf64e8a318b894de88c7
[24/63] notify: api: api types: rename Deleteable* -> Deletable*
        commit: 57ad72daa24afbfcc55e36b512f91011a1087891
[25/63] notify: context: parse builtin config only once
        commit: a9e5bbd91d2a589495fe7d2e9941de1f19b00954
[26/63] notify: context: parse proxy config in trait impl, not at the callsite
        commit: 7fd90635ff9faea47c83be0ec689ab6c7d2c90f4
[27/63] notify: gate sub-modules with the appropriate feature flags
        commit: f7acb38a5cd670b7b1082804698dcd85d9476c32
[28/63] notify: fold mail-forwarder feature into send-impl
        commit: 824ef061666dc4195767cc1940ce6b31dc6ab1b5
[29/63] notify: forwarded mails: don't hardcode severity and metadata fields
        commit: f2974ff716e747e6244108c8355086380ed81979
[30/63] notify: don't pass uid when instantiating forwarded mail notifications
        commit: c9d7c57fbe391830f0aa0cb38d15e5e964862c1f
[31/63] notify: derive PartialEq for Notification type
        commit: 0c274f85220420233f6e608a61ab9c4090c3e227
[32/63] notify: add getters for severity and fields on Notification type
        commit: b57e80c53db004d10ffdbdbf119b7fb2034f8e71
[33/63] notify: add test suite for Notification type
        commit: 38c1bd5e53ad83df709dd7bced3458cee43faf49
[34/63] notify: serialize raw, forwarded mail content as base64
        commit: 19a994bc1205396f083796648602f8439b6f4907
[35/63] notify: unify code paths for legacy-sendmail and regular notifications
        commit: 3e18e96b828aa8a902d2bde0fec9edd97a794649
[36/63] notify: add backwards-compat note to Notification type
        commit: 7022cbe018a0459e3957d2a11203272d9c6ad379
[37/63] notify: remove 'test_target' API
        commit: 516c605451b3b5e9f3230da90517aba5e5d7d494
[38/63] notify: replace notification instantiation with builder pattern
        commit: 6e286d32db7ee38ee4365176d9854a8c2f807407
[39/63] notify: add send receipts
        commit: 8025ce40838af5f56e4867952ba047250fe3e8a7
[40/63] notify: make sure that template name matches SAFE_ID_REGEX
        commit: 5faae51b5308233eed41bca8a18ad9e58116ac90
[41/63] notify: implement notification queue submitter
        commit: b6f89f791e5fdeb94b5befc4d56ac90342870602
[42/63] notify: context: add read_config
        commit: 464812a29f6b703a82379e4a2379d024effe7a10
[43/63] notify: queue: add queue consumer
        commit: 08eb1922dbe279d83852176eeed805034e8ec47d
[44/63] notify: queue: add worker task
        commit: fe26fe0c48363c5a04ef89ee35e849223d4b6ada

[proxmox-backup] applied to git/proxmox-backup.git (master)
[45/63] notifications: don't ignore verification errors in legacy-sendmail mode
        commit: 958e6202a0c343e63be1abe0f50539b86101fadd
[46/63] notifications: adapt to proxmox-notify API changes
        commit: 02abfc8befa4ea5a9e98d18fd8b69b47a6a66fd3
[47/63] api: notification: send test notification via proxmox_notify::send
        commit: 02abfc8befa4ea5a9e98d18fd8b69b47a6a66fd3
[48/63] notifications: send all notifications via proxmox-notify's queue
        commit: 91aa31f58030821c2e05ca941f9e88fec88d2285

[proxmox-mail-forward] applied to git/proxmox-mail-forward.git (master)
[57/63] forward mails via notification queue
        commit: 6e4876c2dd62554c93de66d597ebac4ea527b5b1
[58/63] use return code to signal permanent or temporary failure
        commit: 509984db02d6bed09a740393a9fd9c8ad112f726
[59/63] limit size of mail read from stdin
        commit: c3bf1e9bbbbe6331d26316b028a068a8bcd64d81
[60/63] cargo: move mail forwarder into a workspace member crate
        commit: b49a21da57d748e1405f554eaccf7162b47e5cd3
[61/63] add crate for code shared between the mail-forwarder and the worker
        commit: e1cf6b27c439de75b3d8a894c6fad5f6fed11d35
[62/63] add pve-notify-worker crate and package
        commit: 2b269c9e8c7b1e35fba6bb183587ceec0a2f3806
[63/63] debian: add versioned break for pve-manager and proxmox-backup-server
        commit: 095d3c6668af9a4f67e562f213c193f1186ee7c4

[proxmox-perl-rs] applied to git/proxmox-perl-rs.git (master)
[49/63] pve: notify: adapt to changes in the public API of proxmox-notify
        commit: 0c51204db2bb0b53e3739acf40f4e22069e5eba8
[50/63] pve: notify: add `PVE::RS::Notify::enqueue` method
        commit: 3e40d44a4cdd18b45293dfeffd5d3f7356376716

[pve-cluster] applied to pve-cluster.git (master)
[51/63] notify: send notifications via notification queue
        https://git.proxmox.com/?p=pve-cluster.git;a=commitdiff;h=1a192374cc1f7d63f22e1bace700ea575d367f7b




  parent reply	other threads:[~2026-11-18 21:49 UTC|newest]

Thread overview: 66+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18  9:42 [PATCH many v2 00/63] notify: maintenance, PDM preparations, queue + worker Lukas Wagner
2026-11-18  9:42 ` [PATCH proxmox v2 01/63] notify: disallow `unsafe` code Lukas Wagner
2026-11-18  9:42 ` [PATCH proxmox v2 02/63] notify: refactor error type Lukas Wagner
2026-11-18  9:42 ` [PATCH proxmox v2 03/63] notify: move `Endpoint` trait to endpoint module as pub(crate) Lukas Wagner
2026-11-18  9:42 ` [PATCH proxmox v2 04/63] notify: use ConfigDigest instead of raw byte arrays Lukas Wagner
2026-11-18  9:42 ` [PATCH proxmox v2 05/63] notify: remove static context and pass context as parameter Lukas Wagner
2026-11-18  9:42 ` [PATCH proxmox v2 06/63] notify: matcher: drop unused field matcher verifiers Lukas Wagner
2026-11-18  9:42 ` [PATCH proxmox v2 07/63] notify: introduce 'send-impl', 'api-impl' and 'api-types' features Lukas Wagner
2026-11-18  9:42 ` [PATCH proxmox v2 08/63] notify: matcher: make matcher validation methods public Lukas Wagner
2026-11-18  9:42 ` [PATCH proxmox v2 09/63] notify: matcher: stub CalendarMatcher::matches on wasm32 Lukas Wagner
2026-11-18  9:42 ` [PATCH proxmox v2 10/63] notify: make SendmailEndpoint and SmtpEndpoint private types Lukas Wagner
2026-11-18  9:42 ` [PATCH proxmox v2 11/63] notify: move `send`, `test_target` and `Bus` to new `send` module Lukas Wagner
2026-11-18  9:42 ` [PATCH proxmox v2 12/63] notify: move Origin type to api_types module Lukas Wagner
2026-11-18  9:42 ` [PATCH proxmox v2 13/63] notify: move Target " Lukas Wagner
2026-11-18  9:42 ` [PATCH proxmox v2 14/63] notify: move Config to config module Lukas Wagner
2026-11-18  9:42 ` [PATCH proxmox v2 15/63] notify: move Notification and sub-types to `notification` module Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox v2 16/63] notify: move schema module to api_types Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox v2 17/63] notify: move check_matches from api_types::matcher to send::Bus Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox v2 18/63] notify: move TemplateSource to context module Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox v2 19/63] notify: config: cleanup helper names and visibility Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox v2 20/63] notify: config: rename section config parser/writer methods Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox v2 21/63] notify: config: pass builtins explicitly when parsing section config Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox v2 22/63] notify: implement PartialEq for all Config types Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox v2 23/63] notify: use typed SectionConfigData Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox v2 24/63] notify: api: api types: rename Deleteable* -> Deletable* Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox v2 25/63] notify: context: parse builtin config only once Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox v2 26/63] notify: context: parse proxy config in trait impl, not at the callsite Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox v2 27/63] notify: gate sub-modules with the appropriate feature flags Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox v2 28/63] notify: fold mail-forwarder feature into send-impl Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox v2 29/63] notify: forwarded mails: don't hardcode severity and metadata fields Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox v2 30/63] notify: don't pass uid when instantiating forwarded mail notifications Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox v2 31/63] notify: derive PartialEq for Notification type Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox v2 32/63] notify: add getters for severity and fields on " Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox v2 33/63] notify: add test suite for " Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox v2 34/63] notify: serialize raw, forwarded mail content as base64 Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox v2 35/63] notify: unify code paths for legacy-sendmail and regular notifications Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox v2 36/63] notify: add backwards-compat note to Notification type Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox v2 37/63] notify: remove 'test_target' API Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox v2 38/63] notify: replace notification instantiation with builder pattern Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox v2 39/63] notify: add send receipts Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox v2 40/63] notify: make sure that template name matches SAFE_ID_REGEX Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox v2 41/63] notify: implement notification queue submitter Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox v2 42/63] notify: context: add read_config Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox v2 43/63] notify: queue: add queue consumer Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox v2 44/63] notify: queue: add worker task Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox-backup v2 45/63] notifications: don't ignore verification errors in legacy-sendmail mode Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox-backup v2 46/63] notifications: adapt to proxmox-notify API changes Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox-backup v2 47/63] api: notification: send test notification via proxmox_notify::send Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox-backup v2 48/63] notifications: send all notifications via proxmox-notify's queue Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox-perl-rs v2 49/63] pve: notify: adapt to changes in the public API of proxmox-notify Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox-perl-rs v2 50/63] pve: notify: add `PVE::RS::Notify::enqueue` method Lukas Wagner
2026-11-18  9:43 ` [PATCH pve-cluster v2 51/63] notify: send notifications via notification queue Lukas Wagner
2026-11-18  9:43 ` [PATCH manager v2 52/63] vzdump: send legacy-sendmail notifications via simplified API Lukas Wagner
2026-11-18  9:43 ` [PATCH manager v2 53/63] debian: create notification queue directory via tmpfiles Lukas Wagner
2026-11-18  9:43 ` [PATCH manager v2 54/63] debian: add 'pve-notify-worker' as dependecy Lukas Wagner
2026-11-18  9:43 ` [PATCH manager v2 55/63] api: services: add 'pve-notify-worker' as known service Lukas Wagner
2026-11-18  9:43 ` [PATCH manager v2 56/63] services: request pve-notify-worker.service via Wants Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox-mail-forward v2 57/63] forward mails via notification queue Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox-mail-forward v2 58/63] use return code to signal permanent or temporary failure Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox-mail-forward v2 59/63] limit size of mail read from stdin Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox-mail-forward v2 60/63] cargo: move mail forwarder into a workspace member crate Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox-mail-forward v2 61/63] add crate for code shared between the mail-forwarder and the worker Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox-mail-forward v2 62/63] add pve-notify-worker crate and package Lukas Wagner
2026-11-18  9:43 ` [PATCH proxmox-mail-forward v2 63/63] debian: add versioned break for pve-manager and proxmox-backup-server Lukas Wagner
2026-11-18 21:44 ` Thomas Lamprecht [this message]
2026-11-19 10:45   ` [PATCH many v2 00/63] notify: maintenance, PDM preparations, queue + worker Lukas Wagner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=179503802208.3534257.8475790039140179721.b4-ty@b4 \
    --to=t.lamprecht@proxmox.com \
    --cc=l.wagner@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal