From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 1BF901FF0E3 for ; Wed, 18 Nov 2026 22:49:51 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id D1BBF217BA; Wed, 18 Nov 2026 22:49:47 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com, Lukas Wagner Subject: Re: [PATCH many v2 00/63] notify: maintenance, PDM preparations, queue + worker Date: Wed, 18 Nov 2026 22:44:45 +0100 Message-ID: <179503802208.3534257.8475790039140179721.b4-ty@b4> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261118094348.273646-1-l.wagner@proxmox.com> References: <20261118094348.273646-1-l.wagner@proxmox.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795038582515 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.039 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) KAM_LOTSOFHASH 0.25 Emails with lots of hash-like gibberish RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 7EGGD4NCOYBVW5BZJDHTI3ASY5NGVK7M X-Message-ID-Hash: 7EGGD4NCOYBVW5BZJDHTI3ASY5NGVK7M X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Wed, 18 Nov 2026 10:42:44 +0100, Lukas Wagner wrote: > The notification stack will soon be integrated into a new product (PDM), so it > makes sense to use this opportunity for any pending breaking changes and > general cleanup of the notification stack, partly driven by the worker > implementation, pending feature work (notification history), and overdue > maintenance work to reduce technical debt. > > The introduction of a shared worker implementation lifts long-standing limitations in PVE, > where only the privileged process could send notifications. The worker will also be used > in PDM. > > [...] Applied with some changes on top, thanks! Folded in, besides typos and docs: - worker: notifications failing validation, symlinks and other entries that cannot be opened are marked broken instead of being requeued forever; other read errors after a claim requeue instead of using up an attempt. - moved patch 29 "notify: forwarded mails: don't hardcode severity and metadata fields" before patch 28 "notify: fold mail-forwarder feature into send-impl", so the default feature build works at each commit. - mail-forward: worker unit ordered after pve-cluster, build-dependencies, and the worker package now ships the queue dir tmpfiles entry. dpkg configures it before pve-manager, so the worker started without its queue; patch 53 "debian: create notification queue directory via tmpfiles" got dropped. The worker is uploaded to the PVE repo only. - PBS: patch 47 "api: notification: send test notification via proxmox_notify::send" squashed into patch 46 "notifications: adapt to proxmox-notify API changes", so the latter builds on its own; the multi-line inline tables in Cargo.toml are single-line again, as they need TOML 1.1. Separate commits: - notify: no config read while pmxcfs is not mounted; queue files are 0640 and owned by the submitter, and the queue dir is expected to be 3770, so submitters cannot change or remove each other's notifications (the worker checks the setup at startup); warning if the queue stays locked for five minutes; IDs queued twice are left alone; receipts record 'mailto' for direct email recipients instead of a 'legacy-sendmail' target name. - PBS: the queue dir is fixed up through an O_NOFOLLOW handle instead of refusing to start; old spool files are only renamed, not read as root. - pve-cluster: libpve-notify-perl depends on pve-notify-worker. Left oper open points, input welcome: - retrying deliveries that reached no target. I had a variant reusing the attempt counter, but it seems to fit better with per-target state in the notification history, as then one can ensure nothing gets submitted multiple times for each specific target. - the PVE queue path is defined in pve-rs, mail-forward and the worker; moving PVEContext and the path into a PVE crate would give one source. - the PBS queue sits below the backup-owned state dir, so the backup user can still replace it. - patch 56 "services: request pve-notify-worker.service via Wants" has no rationale; what does the extra Wants= guard against? - mail-forward treats stdin I/O errors like an oversized mail and bounces it. - a pve-docs counterpart for the PBS docs change would be nice [manager] applied to pve-manager.git (master) [52/63] vzdump: send legacy-sendmail notifications via simplified API https://git.proxmox.com/?p=pve-manager.git;a=commitdiff;h=ba8dd37eb7012ad837dba1831da23e964ee71d63 [53/63] debian: create notification queue directory via tmpfiles (no commit info) [54/63] debian: add 'pve-notify-worker' as dependecy https://git.proxmox.com/?p=pve-manager.git;a=commitdiff;h=fe4f131c084a0e077a55678ca1929b1d236db709 [55/63] api: services: add 'pve-notify-worker' as known service https://git.proxmox.com/?p=pve-manager.git;a=commitdiff;h=7f87d76b293ecd20ba983e43ac41cd941eb9d9fd [56/63] services: request pve-notify-worker.service via Wants https://git.proxmox.com/?p=pve-manager.git;a=commitdiff;h=618c236fb407d6204980197f8eb0d9d31d2dab5e [proxmox] applied to git/proxmox.git (master) [01/63] notify: disallow `unsafe` code commit: 8c715db3a42b1ef996784dfbcfde326a5d9a0116 [02/63] notify: refactor error type commit: c00b7d2b001af30ed00ab78e6646cb4dab26f104 [03/63] notify: move `Endpoint` trait to endpoint module as pub(crate) commit: 089c815ba6d7ce2c4ac516d2f73c5b3905a71260 [04/63] notify: use ConfigDigest instead of raw byte arrays commit: f37cfe258c2e741f4936d437c233ff2b95e3ab7f [05/63] notify: remove static context and pass context as parameter commit: 3066b4436db4198445785b733cd3e77c9944381b [06/63] notify: matcher: drop unused field matcher verifiers commit: a9c8845e2b4eb307ec528087909df2098541ed68 [07/63] notify: introduce 'send-impl', 'api-impl' and 'api-types' features commit: c0d14243fe83f18c8dadf22638daf304ea5a2a16 [08/63] notify: matcher: make matcher validation methods public commit: ed684af62fa34af8e49a10086349e62550a0f258 [09/63] notify: matcher: stub CalendarMatcher::matches on wasm32 commit: a7ea27438b0d1246db109663c45549a926f16051 [10/63] notify: make SendmailEndpoint and SmtpEndpoint private types commit: cafaa7ad378188ee03ef3e8c4fadd1c31867dbe5 [11/63] notify: move `send`, `test_target` and `Bus` to new `send` module commit: 172e40456be2b21534458c4ce262746bb915b45d [12/63] notify: move Origin type to api_types module commit: 6664b447f484c6608d536874deceff9d4a4e903b [13/63] notify: move Target type to api_types module commit: 2dcca96b0f2e19518f1baa8b84d680a28b638d22 [14/63] notify: move Config to config module commit: 39cda59d301b16b6d5cb2fac3cf1fcd3bb925256 [15/63] notify: move Notification and sub-types to `notification` module commit: 66edfc9b337737f7add3ea1c24de43319586d816 [16/63] notify: move schema module to api_types commit: b9837a3f02c1055dacd5e0b1d7e9922cc7a026be [17/63] notify: move check_matches from api_types::matcher to send::Bus commit: d6973a53b8737978b88bf2f6fb5f6a6fd3aeb0e3 [18/63] notify: move TemplateSource to context module commit: bc95b07be0bde600bb14defa75a7142820da3537 [19/63] notify: config: cleanup helper names and visibility commit: ef6e079d1d45896ed05ec37b51dac5aadf1a7eb3 [20/63] notify: config: rename section config parser/writer methods commit: bd240427b97f3a00e4feda3e1c6f1fbe15c6e322 [21/63] notify: config: pass builtins explicitly when parsing section config commit: 13c6f06ca0ae481c900f8c11c4260386201e0182 [22/63] notify: implement PartialEq for all Config types commit: b59510fda0ce184d784d805004ebd8e84a49f15f [23/63] notify: use typed SectionConfigData commit: 79b95f7c2b5236080571bf64e8a318b894de88c7 [24/63] notify: api: api types: rename Deleteable* -> Deletable* commit: 57ad72daa24afbfcc55e36b512f91011a1087891 [25/63] notify: context: parse builtin config only once commit: a9e5bbd91d2a589495fe7d2e9941de1f19b00954 [26/63] notify: context: parse proxy config in trait impl, not at the callsite commit: 7fd90635ff9faea47c83be0ec689ab6c7d2c90f4 [27/63] notify: gate sub-modules with the appropriate feature flags commit: f7acb38a5cd670b7b1082804698dcd85d9476c32 [28/63] notify: fold mail-forwarder feature into send-impl commit: 824ef061666dc4195767cc1940ce6b31dc6ab1b5 [29/63] notify: forwarded mails: don't hardcode severity and metadata fields commit: f2974ff716e747e6244108c8355086380ed81979 [30/63] notify: don't pass uid when instantiating forwarded mail notifications commit: c9d7c57fbe391830f0aa0cb38d15e5e964862c1f [31/63] notify: derive PartialEq for Notification type commit: 0c274f85220420233f6e608a61ab9c4090c3e227 [32/63] notify: add getters for severity and fields on Notification type commit: b57e80c53db004d10ffdbdbf119b7fb2034f8e71 [33/63] notify: add test suite for Notification type commit: 38c1bd5e53ad83df709dd7bced3458cee43faf49 [34/63] notify: serialize raw, forwarded mail content as base64 commit: 19a994bc1205396f083796648602f8439b6f4907 [35/63] notify: unify code paths for legacy-sendmail and regular notifications commit: 3e18e96b828aa8a902d2bde0fec9edd97a794649 [36/63] notify: add backwards-compat note to Notification type commit: 7022cbe018a0459e3957d2a11203272d9c6ad379 [37/63] notify: remove 'test_target' API commit: 516c605451b3b5e9f3230da90517aba5e5d7d494 [38/63] notify: replace notification instantiation with builder pattern commit: 6e286d32db7ee38ee4365176d9854a8c2f807407 [39/63] notify: add send receipts commit: 8025ce40838af5f56e4867952ba047250fe3e8a7 [40/63] notify: make sure that template name matches SAFE_ID_REGEX commit: 5faae51b5308233eed41bca8a18ad9e58116ac90 [41/63] notify: implement notification queue submitter commit: b6f89f791e5fdeb94b5befc4d56ac90342870602 [42/63] notify: context: add read_config commit: 464812a29f6b703a82379e4a2379d024effe7a10 [43/63] notify: queue: add queue consumer commit: 08eb1922dbe279d83852176eeed805034e8ec47d [44/63] notify: queue: add worker task commit: fe26fe0c48363c5a04ef89ee35e849223d4b6ada [proxmox-backup] applied to git/proxmox-backup.git (master) [45/63] notifications: don't ignore verification errors in legacy-sendmail mode commit: 958e6202a0c343e63be1abe0f50539b86101fadd [46/63] notifications: adapt to proxmox-notify API changes commit: 02abfc8befa4ea5a9e98d18fd8b69b47a6a66fd3 [47/63] api: notification: send test notification via proxmox_notify::send commit: 02abfc8befa4ea5a9e98d18fd8b69b47a6a66fd3 [48/63] notifications: send all notifications via proxmox-notify's queue commit: 91aa31f58030821c2e05ca941f9e88fec88d2285 [proxmox-mail-forward] applied to git/proxmox-mail-forward.git (master) [57/63] forward mails via notification queue commit: 6e4876c2dd62554c93de66d597ebac4ea527b5b1 [58/63] use return code to signal permanent or temporary failure commit: 509984db02d6bed09a740393a9fd9c8ad112f726 [59/63] limit size of mail read from stdin commit: c3bf1e9bbbbe6331d26316b028a068a8bcd64d81 [60/63] cargo: move mail forwarder into a workspace member crate commit: b49a21da57d748e1405f554eaccf7162b47e5cd3 [61/63] add crate for code shared between the mail-forwarder and the worker commit: e1cf6b27c439de75b3d8a894c6fad5f6fed11d35 [62/63] add pve-notify-worker crate and package commit: 2b269c9e8c7b1e35fba6bb183587ceec0a2f3806 [63/63] debian: add versioned break for pve-manager and proxmox-backup-server commit: 095d3c6668af9a4f67e562f213c193f1186ee7c4 [proxmox-perl-rs] applied to git/proxmox-perl-rs.git (master) [49/63] pve: notify: adapt to changes in the public API of proxmox-notify commit: 0c51204db2bb0b53e3739acf40f4e22069e5eba8 [50/63] pve: notify: add `PVE::RS::Notify::enqueue` method commit: 3e40d44a4cdd18b45293dfeffd5d3f7356376716 [pve-cluster] applied to pve-cluster.git (master) [51/63] notify: send notifications via notification queue https://git.proxmox.com/?p=pve-cluster.git;a=commitdiff;h=1a192374cc1f7d63f22e1bace700ea575d367f7b