* [PATCH docs] notifications: document notification queue and mail forwarding limits
@ 2026-11-19 10:07 Lukas Wagner
0 siblings, 0 replies; only message in thread
From: Lukas Wagner @ 2026-11-19 10:07 UTC (permalink / raw)
To: pve-devel
Notifications are now queued and handled by pve-notify-worker, which
warrants a new section. The migration to a worker changed how
logging works for notifications, so explain that as well.
The mail forwarder has slightly changed: if PVE is co-installed with
PBS, the mail is now only fed into PVE's queue instead of both. Also
document the size cap for forwarded mails.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
---
notifications.adoc | 50 +++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 49 insertions(+), 1 deletion(-)
diff --git a/notifications.adoc b/notifications.adoc
index 5712794..5aef22a 100644
--- a/notifications.adoc
+++ b/notifications.adoc
@@ -495,8 +495,51 @@ NOTE: Backup job notifications only have `job-id` set if the backup job
was executed automatically based on its schedule, but not if it was triggered
manually by the 'Run now' button in the UI.
+[[notification_queue]]
+Notification Queue
+------------------
+
+Notification events are not sent directly by the process that emits them.
+Instead, they are put into a local queue and sent by the `pve-notify-worker`
+service within a few seconds. Errors while sending, for example an unreachable
+mail server, are therefore logged to the system journal of that service, not to
+the task log of the job that emitted the event. Test notifications sent from
+the GUI are the exception: these are sent directly, so that the result can be
+shown.
+
+Logs for the `pve-notify-worker` service can be viewed in the {pve} Web UI
+under the *System* tab for the node that emitted the notification event, by
+selecting the `pve-notify-worker` service and clicking the *Service System Log*
+button. Alternatively, run `journalctl -u pve-notify-worker` on that node.
+
+If the service is shut down gracefully while sending a notification (e.g. due
+to a `systemctl stop pve-notify-worker`), the service attempts to finish
+sending before shutting down. If the service is stopped irregularly, e.g. due
+to a crash or being killed, it sends the notification again after it has been
+restarted. There can be up to 2 retries after the initial attempt, after which
+the notification is dropped by the worker. Targets that have already been
+notified during a prior attempt may be notified again. The retries only cover
+interruptions of the service. Failed deliveries to individual targets, for
+example due to an unreachable email server, are not retried.
+
+The local notification queue is located at
+`/var/lib/pve-manager/notifications/queue`, with each queued notification being
+a file in this directory. Notification files follow the naming scheme
+`notification-<uuid>` with an optional appended attempt counter, for instance
+`notification-46516f7e-7eb7-43e7-b776-6048f144a6fe` for a freshly queued
+notification or `notification-46516f7e-7eb7-43e7-b776-6048f144a6fe.attempt-1`
+for a notification that is either currently being handled or was interrupted
+during the first attempt.
+
+Generally, this directory should be empty if all queued notifications have been
+sent. Notification files with a `.broken` suffix indicate that the file could
+not be successfully handled by the `pve-notify-worker` service, either due to
+invalid contents or other unexpected properties. The service appends the suffix
+to allow further investigation by the administrator. It is safe to remove
+`.broken` files at any moment.
+
System Mail Forwarding
----------------------
+----------------------
Certain local system daemons, such as `smartd`, generate notification emails
that are initially directed to the local `root` user. These mails are converted
@@ -509,6 +552,11 @@ the system tries to extract both a subject line and the main text body
from the email content. In instances where emails solely consist of HTML
content, they will be transformed into plain text format during this process.
+If Proxmox Backup Server is installed on the same host, system mails are only
+handled by the notification system of {pve}. Mails larger than 1 MiB are
+rejected and bounce. If a mail cannot be queued, the local mail system keeps it
+and tries again later.
+
Permissions
-----------
--
2.47.3
^ permalink raw reply related [flat|nested] only message in thread
only message in thread, other threads:[~2026-11-19 10:07 UTC | newest]
Thread overview: (only message) (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-19 10:07 [PATCH docs] notifications: document notification queue and mail forwarding limits Lukas Wagner
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox