From: Lukas Wagner <l.wagner@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [pve-devel] [PATCH v4 pve-manager 47/69] api: replication: send notifications via new notification module
Date: Thu, 20 Jul 2023 16:32:14 +0200 [thread overview]
Message-ID: <20230720143236.652292-48-l.wagner@proxmox.com> (raw)
In-Reply-To: <20230720143236.652292-1-l.wagner@proxmox.com>
If the new 'target-replication' option in datacenter.cfg is set
to a notification target, we send notifications that way.
If it is not set, we continue send a notification to the
default target (mail to root@pam).
There is also a new 'replication' option. It controls whether
to send a notification at all.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
---
PVE/API2/Replication.pm | 63 ++++++++++++++++++++++++++++-------------
1 file changed, 43 insertions(+), 20 deletions(-)
diff --git a/PVE/API2/Replication.pm b/PVE/API2/Replication.pm
index 89c5a802..d61518ba 100644
--- a/PVE/API2/Replication.pm
+++ b/PVE/API2/Replication.pm
@@ -15,6 +15,7 @@ use PVE::QemuConfig;
use PVE::QemuServer;
use PVE::LXC::Config;
use PVE::LXC;
+use PVE::Notify;
use PVE::RESTHandler;
@@ -91,6 +92,24 @@ my sub _should_mail_at_failcount {
return $i * 48 == $fail_count;
};
+my $replication_error_subject_template = "Replication Job: '{{job-id}}' failed";
+my $replication_error_body_template = <<EOT;
+{{#verbatim}}
+Replication job '{{job-id}}' with target '{{job-target}}' and schedule '{{job-schedule}}' failed!
+
+Last successful sync: {{timestamp last-sync}}
+Next sync try: {{timestamp next-sync}}
+Failure count: {{failure-count}}
+
+{{#if (eq failure-count 3)}}
+Note: The system will now reduce the frequency of error reports, as the job
+appears to be stuck.
+{{/if}}
+Error:
+{{verbatim-monospaced error}}
+{{/verbatim}}
+EOT
+
my sub _handle_job_err {
my ($job, $err, $mail) = @_;
@@ -103,33 +122,37 @@ my sub _handle_job_err {
return if !_should_mail_at_failcount($fail_count);
- my $schedule = $job->{schedule} // '*/15';
-
- my $msg = "Replication job $job->{id} with target '$job->{target}' and schedule";
- $msg .= " '$schedule' failed!\n";
-
- $msg .= " Last successful sync: ";
- if (my $last_sync = $jobstate->{last_sync}) {
- $msg .= render_timestamp($last_sync) ."\n";
- } else {
- $msg .= "None/Unknown\n";
- }
# not yet updated, so $job->next_sync here is actually the current one.
# NOTE: Copied from PVE::ReplicationState::job_status()
my $next_sync = $job->{next_sync} + 60 * ($fail_count <= 3 ? 5 * $fail_count : 30);
- $msg .= " Next sync try: " . render_timestamp($next_sync) ."\n";
- $msg .= " Failure count: $fail_count\n";
-
- if ($fail_count == 3) {
- $msg .= "\nNote: The system will now reduce the frequency of error reports,";
- $msg .= " as the job appears to be stuck.\n";
- }
+ # The replication job is run every 15 mins if no schedule is set.
+ my $schedule = $job->{schedule} // '*/15';
- $msg .= "\nError:\n$err";
+ my $properties = {
+ "failure-count" => $fail_count,
+ "last-sync" => $jobstate->{last_sync},
+ "next-sync" => $next_sync,
+ "job-id" => $job->{id},
+ "job-target" => $job->{target},
+ "job-schedule" => $schedule,
+ "error" => $err,
+ };
eval {
- PVE::Tools::sendmail('root', "Replication Job: $job->{id} failed", $msg)
+ my $dcconf = PVE::Cluster::cfs_read_file('datacenter.cfg');
+ my $target = $dcconf->{notify}->{'target-replication'} // PVE::Notify::default_target();
+ my $notify = $dcconf->{notify}->{'replication'} // 'always';
+
+ if ($notify eq 'always') {
+ PVE::Notify::error(
+ $target,
+ $replication_error_subject_template,
+ $replication_error_body_template,
+ $properties
+ );
+ }
+
};
warn ": $@" if $@;
}
--
2.39.2
next prev parent reply other threads:[~2023-07-20 14:33 UTC|newest]
Thread overview: 84+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-07-20 14:31 [pve-devel] [PATCH v4 many 00/69] fix #4156: introduce new notification system Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox 01/69] section-config: derive Clone for SectionConfigData Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox 02/69] schema: add schema/format for comments Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox 03/69] add proxmox-notify crate Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox 04/69] notify: preparation for the first endpoint plugin Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox 05/69] notify: preparation for the API Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox 06/69] notify: api: add API for sending notifications/testing endpoints Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox 07/69] notify: add sendmail plugin Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox 08/69] notify: api: add API for sendmail endpoints Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox 09/69] notify: add gotify endpoint Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox 10/69] notify: api: add API for gotify endpoints Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox 11/69] notify: add notification groups Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox 12/69] notify: api: add API for groups Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox 13/69] notify: add notification filter mechanism Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox 14/69] notify: api: add API for filters Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox 15/69] notify: add template rendering Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox 16/69] notify: add example for " Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox 17/69] notify: add context Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox 18/69] notify: sendmail: allow users as recipients Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox 19/69] notify: sendmail: query default author/mailfrom from context Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox 20/69] notify: gotify: add proxy support Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox 21/69] notify: api: allow to query entities referenced by filter/target Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox 22/69] notify: on deletion, check if a filter/endp. is still used by anything Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox 23/69] notify: ensure that filter/group/endpoint names are unique Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox 24/69] notify: additional logging when sending a notification Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox 25/69] notify: add debian packaging Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox-perl-rs 26/69] add PVE::RS::Notify module Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox-perl-rs 27/69] notify: add api for sending notifications/testing endpoints Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox-perl-rs 28/69] notify: add api for notification groups Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox-perl-rs 29/69] notify: add api for sendmail endpoints Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox-perl-rs 30/69] notify: add api for gotify endpoints Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox-perl-rs 31/69] notify: add api for notification filters Lukas Wagner
2023-07-20 14:31 ` [pve-devel] [PATCH v4 proxmox-perl-rs 32/69] notify: sendmail: support the `mailto-user` parameter Lukas Wagner
2023-07-20 14:32 ` [pve-devel] [PATCH v4 proxmox-perl-rs 33/69] notify: implement context for getting default author/mailfrom Lukas Wagner
2023-07-20 14:32 ` [pve-devel] [PATCH v4 proxmox-perl-rs 34/69] notify: add context for getting http_proxy from datacenter.cfg Lukas Wagner
2023-07-20 14:32 ` [pve-devel] [PATCH v4 proxmox-perl-rs 35/69] notify: add wrapper for `get_referenced_entities` Lukas Wagner
2023-07-20 14:32 ` [pve-devel] [PATCH v4 pve-cluster 36/69] cluster files: add notifications.cfg Lukas Wagner
2023-07-24 13:02 ` [pve-devel] partially-applied: " Wolfgang Bumiller
2023-07-20 14:32 ` [pve-devel] [PATCH v4 pve-cluster 37/69] datacenter: add APT/fencing/replication notification configuration Lukas Wagner
2023-07-20 14:32 ` [pve-devel] [PATCH v4 pve-cluster 38/69] add libpve-notify-perl package Lukas Wagner
2023-07-20 14:32 ` [pve-devel] [PATCH v4 pve-guest-common 39/69] vzdump: add config options for new notification backend Lukas Wagner
2023-07-24 13:30 ` [pve-devel] partially-applied: " Wolfgang Bumiller
2023-07-20 14:32 ` [pve-devel] [PATCH v4 pve-common 40/69] JSONSchema: increase maxLength of config-digest to 64 Lukas Wagner
2023-07-24 9:56 ` [pve-devel] applied: " Wolfgang Bumiller
2023-07-20 14:32 ` [pve-devel] [PATCH v4 pve-ha-manager 41/69] manager: send notifications via new notification module Lukas Wagner
2023-07-20 14:32 ` [pve-devel] [PATCH v4 pve-manager 42/69] test: fix names of .PHONY targets Lukas Wagner
2023-07-24 13:59 ` Wolfgang Bumiller
2023-07-20 14:32 ` [pve-devel] [PATCH v4 pve-manager 43/69] d/control: add dependency to `libpve-notify-perl` Lukas Wagner
2023-07-20 14:32 ` [pve-devel] [PATCH v4 pve-manager 44/69] vzdump: send notifications via new notification module Lukas Wagner
2023-07-20 14:32 ` [pve-devel] [PATCH v4 pve-manager 45/69] test: rename mail_test.pl to vzdump_notification_test.pl Lukas Wagner
2023-07-20 14:32 ` [pve-devel] [PATCH v4 pve-manager 46/69] api: apt: send notification via new notification module Lukas Wagner
2023-07-20 14:32 ` Lukas Wagner [this message]
2023-07-20 14:32 ` [pve-devel] [PATCH v4 pve-manager 48/69] api: prepare api handler module for notification config Lukas Wagner
2023-07-20 14:32 ` [pve-devel] [PATCH v4 pve-manager 49/69] api: notification: add api routes for groups Lukas Wagner
2023-07-24 13:54 ` Wolfgang Bumiller
2023-07-20 14:32 ` [pve-devel] [PATCH v4 pve-manager 50/69] api: notification: add api routes for sendmail endpoints Lukas Wagner
2023-07-24 13:55 ` Wolfgang Bumiller
2023-07-20 14:32 ` [pve-devel] [PATCH v4 pve-manager 51/69] api: notification: add api routes for gotify endpoints Lukas Wagner
2023-07-24 13:55 ` Wolfgang Bumiller
2023-07-25 9:02 ` Thomas Lamprecht
2023-07-20 14:32 ` [pve-devel] [PATCH v4 pve-manager 52/69] api: notification: add api routes for filters Lukas Wagner
2023-07-24 13:56 ` Wolfgang Bumiller
2023-07-20 14:32 ` [pve-devel] [PATCH v4 pve-manager 53/69] api: notification: allow fetching notification targets Lukas Wagner
2023-07-20 14:32 ` [pve-devel] [PATCH v4 pve-manager 54/69] api: notification: allow to test targets Lukas Wagner
2023-07-20 14:32 ` [pve-devel] [PATCH v4 pve-manager 55/69] api: notification: disallow removing targets if they are used Lukas Wagner
2023-07-24 13:50 ` Wolfgang Bumiller
2023-07-24 14:09 ` Lukas Wagner
2023-07-20 14:32 ` [pve-devel] [PATCH v4 pve-manager 56/69] ui: backup: allow to select notification target for jobs Lukas Wagner
2023-07-20 14:32 ` [pve-devel] [PATCH v4 pve-manager 57/69] ui: backup: adapt backup job details to new notification params Lukas Wagner
2023-07-20 14:32 ` [pve-devel] [PATCH v4 pve-manager 58/69] ui: backup: allow to set notification-target for one-off backups Lukas Wagner
2023-07-20 14:32 ` [pve-devel] [PATCH v4 pve-manager 59/69] ui: allow to configure notification event -> target mapping Lukas Wagner
2023-07-20 14:32 ` [pve-devel] [PATCH v4 pve-manager 60/69] ui: add notification target configuration panel Lukas Wagner
2023-07-20 14:32 ` [pve-devel] [PATCH v4 pve-manager 61/69] ui: perm path: add ACL paths for notifications, usb and pci mappings Lukas Wagner
2023-07-20 14:32 ` [pve-devel] [PATCH v4 pve-manager 62/69] ui: perm path: increase width of the perm path selector combobox Lukas Wagner
2023-07-20 14:32 ` [pve-devel] [PATCH v4 pve-manager 63/69] ui: dc: remove notify key from datacenter option view Lukas Wagner
2023-07-20 14:32 ` [pve-devel] [PATCH v4 proxmox-widget-toolkit 64/69] notification: add gui for sendmail notification endpoints Lukas Wagner
2023-07-20 14:32 ` [pve-devel] [PATCH v4 proxmox-widget-toolkit 65/69] notification: add gui for gotify " Lukas Wagner
2023-07-20 14:32 ` [pve-devel] [PATCH v4 proxmox-widget-toolkit 66/69] notification: add gui for notification groups Lukas Wagner
2023-07-20 14:32 ` [pve-devel] [PATCH v4 proxmox-widget-toolkit 67/69] notification: allow to select filter for notification targets Lukas Wagner
2023-07-20 14:32 ` [pve-devel] [PATCH v4 proxmox-widget-toolkit 68/69] notification: add ui for managing notification filters Lukas Wagner
2023-07-20 14:32 ` [pve-devel] [PATCH v4 pve-docs 69/69] add documentation for the new notification system Lukas Wagner
2023-07-24 9:07 ` [pve-devel] partially-applied: [PATCH v4 many 00/69] fix #4156: introduce " Wolfgang Bumiller
2023-07-24 9:20 ` [pve-devel] " Thomas Lamprecht
2023-07-24 9:23 ` [pve-devel] partially-applied: " Wolfgang Bumiller
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230720143236.652292-48-l.wagner@proxmox.com \
--to=l.wagner@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox