From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 647581FF0AF for ; Fri, 20 Nov 2026 09:40:49 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 5F9E721685; Fri, 20 Nov 2026 09:40:48 +0100 (CET) From: Lukas Wagner To: pbs-devel@lists.proxmox.com Subject: [PATCH proxmox-backup] docs: notifications: expand notification queue documentation Date: Fri, 20 Nov 2026 09:40:40 +0100 Message-ID: <20261120084042.94455-1-l.wagner@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795164044091 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.186 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 5NRROMVNGYJJMOJ6SUT6JGUYTOFMIM6O X-Message-ID-Hash: 5NRROMVNGYJJMOJ6SUT6JGUYTOFMIM6O X-MailFrom: l.wagner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Move the description of queued delivery out of the overview into its own section and extend it, mirroring the Proxmox VE documentation. Point out where to find the logs of the proxmox-backup service in the GUI, how interrupted deliveries are retried and that this can lead to duplicate notifications for some targets, and describe the queue directory, including the naming scheme of queued files and what files with a '.broken' suffix mean. Signed-off-by: Lukas Wagner --- docs/notifications.rst | 53 ++++++++++++++++++++++++++++++++++-------- 1 file changed, 43 insertions(+), 10 deletions(-) diff --git a/docs/notifications.rst b/docs/notifications.rst index 1355e2f17..621e3e872 100644 --- a/docs/notifications.rst +++ b/docs/notifications.rst @@ -17,16 +17,6 @@ Overview is routed to by a matcher. There are multiple types of target, mail-based (Sendmail and SMTP) and Gotify. -Notification events are not sent by the process that emits them. They are put -into a queue and sent by the ``proxmox-backup`` service within a few seconds. -Errors while sending, for example an unreachable mail server, are therefore -logged to the system journal of that service, see -``journalctl -u proxmox-backup``, not to the task log of the job that emitted -the event. Test notifications sent from the GUI are the exception, they are -sent directly, so that the result can be shown. If the service is interrupted -while sending a notification, it sends the notification again after it has been -restarted. - Global notification settings can be configured in the GUI under *Configuration → Notifications*. The configuration is stored in :ref:`notifications.cfg` and :ref:`notifications_priv.cfg` - the latter contains sensitive configuration @@ -447,6 +437,49 @@ Metadata field Description .. NOTE:: The daily task checking for any available system updates only sends notifications if the node has an active subscription. +.. _notification_queue: + +Notification Queue +------------------ + +Notification events are not sent directly by the process that emits them. +Instead, they are put into a local queue and sent by the ``proxmox-backup`` +service within a few seconds. Errors while sending, for example an unreachable +mail server, are therefore logged to the system journal of that service, not to +the task log of the job that emitted the event. Test notifications sent from the +GUI are the exception: these are sent directly, so that the result can be shown. + +Logs for the ``proxmox-backup`` service can be viewed in the GUI under +*Administration → Services*, by selecting the ``proxmox-backup`` service and +clicking the *Service System Log* button. Alternatively, run +``journalctl -u proxmox-backup``. + +If the service is shut down gracefully while sending a notification (e.g. due to +a ``systemctl stop proxmox-backup``), the service attempts to finish sending +before shutting down. If the service is stopped irregularly, e.g. due to a crash +or being killed, it sends the notification again after it has been restarted. +There can be up to 2 retries after the initial attempt, after which the +notification is dropped. Targets that have already been notified during a prior +attempt may be notified again. The retries only cover interruptions of the +service. Failed deliveries to individual targets, for example due to an +unreachable email server, are not retried. + +The local notification queue is located at +``/var/lib/proxmox-backup/notifications/queue``, with each queued notification +being a file in this directory. Notification files follow the naming scheme +``notification-`` with an optional appended attempt counter, for instance +``notification-46516f7e-7eb7-43e7-b776-6048f144a6fe`` for a freshly queued +notification or ``notification-46516f7e-7eb7-43e7-b776-6048f144a6fe.attempt-1`` +for a notification that is either currently being handled or was interrupted +during the first attempt. + +Generally, this directory should be empty if all queued notifications have been +sent. Notification files with a ``.broken`` suffix indicate that the file could +not be successfully handled by the ``proxmox-backup`` service, either due to +invalid contents or other unexpected properties. The service appends the suffix +to allow further investigation by the administrator. It is safe to remove +``.broken`` files at any moment. + System Mail Forwarding ---------------------- Certain local system daemons, such as ``smartd``, send notification emails to -- 2.47.3