From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 2454E1FF0DF for ; Thu, 19 Nov 2026 11:07:53 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 66EB7216B3; Thu, 19 Nov 2026 11:07:51 +0100 (CET) From: Lukas Wagner To: pve-devel@lists.proxmox.com Subject: [PATCH docs] notifications: document notification queue and mail forwarding limits Date: Thu, 19 Nov 2026 11:07:43 +0100 Message-ID: <20261119100743.238781-1-l.wagner@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795082867205 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.187 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: MPBDJVHSPGJZDD4NNHT7XLGHOVZOVXV2 X-Message-ID-Hash: MPBDJVHSPGJZDD4NNHT7XLGHOVZOVXV2 X-MailFrom: l.wagner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Notifications are now queued and handled by pve-notify-worker, which warrants a new section. The migration to a worker changed how logging works for notifications, so explain that as well. The mail forwarder has slightly changed: if PVE is co-installed with PBS, the mail is now only fed into PVE's queue instead of both. Also document the size cap for forwarded mails. Signed-off-by: Lukas Wagner --- notifications.adoc | 50 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 49 insertions(+), 1 deletion(-) diff --git a/notifications.adoc b/notifications.adoc index 5712794..5aef22a 100644 --- a/notifications.adoc +++ b/notifications.adoc @@ -495,8 +495,51 @@ NOTE: Backup job notifications only have `job-id` set if the backup job was executed automatically based on its schedule, but not if it was triggered manually by the 'Run now' button in the UI. +[[notification_queue]] +Notification Queue +------------------ + +Notification events are not sent directly by the process that emits them. +Instead, they are put into a local queue and sent by the `pve-notify-worker` +service within a few seconds. Errors while sending, for example an unreachable +mail server, are therefore logged to the system journal of that service, not to +the task log of the job that emitted the event. Test notifications sent from +the GUI are the exception: these are sent directly, so that the result can be +shown. + +Logs for the `pve-notify-worker` service can be viewed in the {pve} Web UI +under the *System* tab for the node that emitted the notification event, by +selecting the `pve-notify-worker` service and clicking the *Service System Log* +button. Alternatively, run `journalctl -u pve-notify-worker` on that node. + +If the service is shut down gracefully while sending a notification (e.g. due +to a `systemctl stop pve-notify-worker`), the service attempts to finish +sending before shutting down. If the service is stopped irregularly, e.g. due +to a crash or being killed, it sends the notification again after it has been +restarted. There can be up to 2 retries after the initial attempt, after which +the notification is dropped by the worker. Targets that have already been +notified during a prior attempt may be notified again. The retries only cover +interruptions of the service. Failed deliveries to individual targets, for +example due to an unreachable email server, are not retried. + +The local notification queue is located at +`/var/lib/pve-manager/notifications/queue`, with each queued notification being +a file in this directory. Notification files follow the naming scheme +`notification-` with an optional appended attempt counter, for instance +`notification-46516f7e-7eb7-43e7-b776-6048f144a6fe` for a freshly queued +notification or `notification-46516f7e-7eb7-43e7-b776-6048f144a6fe.attempt-1` +for a notification that is either currently being handled or was interrupted +during the first attempt. + +Generally, this directory should be empty if all queued notifications have been +sent. Notification files with a `.broken` suffix indicate that the file could +not be successfully handled by the `pve-notify-worker` service, either due to +invalid contents or other unexpected properties. The service appends the suffix +to allow further investigation by the administrator. It is safe to remove +`.broken` files at any moment. + System Mail Forwarding ---------------------- +---------------------- Certain local system daemons, such as `smartd`, generate notification emails that are initially directed to the local `root` user. These mails are converted @@ -509,6 +552,11 @@ the system tries to extract both a subject line and the main text body from the email content. In instances where emails solely consist of HTML content, they will be transformed into plain text format during this process. +If Proxmox Backup Server is installed on the same host, system mails are only +handled by the notification system of {pve}. Mails larger than 1 MiB are +rejected and bounce. If a mail cannot be queued, the local mail system keeps it +and tries again later. + Permissions ----------- -- 2.47.3