public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Fabian Ebner <f.ebner@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [pve-devel] [PATCH manager] api: replication: don't send mails about failed replication only once
Date: Fri, 22 Apr 2022 14:15:48 +0200	[thread overview]
Message-ID: <20220422121548.69146-1-f.ebner@proxmox.com> (raw)

but rather multiple times becoming exponentially less frequent.

Suggested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
 PVE/API2/Replication.pm | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/PVE/API2/Replication.pm b/PVE/API2/Replication.pm
index a1bcd89b..a7e54497 100644
--- a/PVE/API2/Replication.pm
+++ b/PVE/API2/Replication.pm
@@ -74,6 +74,19 @@ sub run_single_job {
 sub run_jobs {
     my ($now, $logfunc, $verbose, $mail) = @_;
 
+    my $mail_at_fail_count = sub {
+	my ($fail_count) = @_;
+
+	return 1 if $fail_count == 1;
+
+	# failing job is re-tried every half hour, try to send one mail after 1, 2, 4, 8, etc. days
+	my $i = 1;
+	while ($i * 48 < $fail_count) {
+	    $i = $i * 2;
+	}
+	return $i * 48 == $fail_count;
+    };
+
     my $iteration = $now // time();
 
     my $code = sub {
@@ -93,7 +106,7 @@ sub run_jobs {
 		my $jobstate = PVE::ReplicationState::extract_job_state($state, $jobcfg);
 		eval {
 		    PVE::Tools::sendmail('root', "Replication Job: $jobcfg->{id} failed", $err)
-			if $jobstate->{fail_count} == 1 && $mail;
+			if $mail && $mail_at_fail_count->($jobstate->{fail_count});
 		};
 		warn ": $@" if $@;
 	    }
-- 
2.30.2





             reply	other threads:[~2022-04-22 12:16 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-04-22 12:15 Fabian Ebner [this message]
2022-04-27  8:30 ` [pve-devel] applied: " Thomas Lamprecht

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20220422121548.69146-1-f.ebner@proxmox.com \
    --to=f.ebner@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal