[pbs-devel] [PATCH proxmox-backup 1/3] tools/systemd/time: implement some Traits for TimeSpan

[pbs-devel] [PATCH proxmox-backup 1/3] tools/systemd/time: implement some Traits for TimeSpan

[Dominik Csapak]

namely
* From<Duration> (to convert easily from duration to timespan)
* Display (for better formatting)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
if wanted, we can optimize the display trait a bit further, e.g.
only showing the biggest two units instead

 src/tools/systemd/time.rs | 84 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 84 insertions(+)

diff --git a/src/tools/systemd/time.rs b/src/tools/systemd/time.rs
index 7cc42415..75fb0ea2 100644
--- a/src/tools/systemd/time.rs
+++ b/src/tools/systemd/time.rs
@@ -141,6 +141,90 @@ impl From<TimeSpan> for f64 {
     }
 }
 
+impl From<std::time::Duration> for TimeSpan {
+    fn from(duration: std::time::Duration) -> Self {
+        let mut duration = duration.as_nanos();
+        let nsec = (duration % 1000) as u64;
+        duration /= 1000;
+        let usec = (duration % 1000) as u64;
+        duration /= 1000;
+        let msec = (duration % 1000) as u64;
+        duration /= 1000;
+        let seconds = (duration % 60) as u64;
+        duration /= 60;
+        let minutes = (duration % 60) as u64;
+        duration /= 60;
+        let hours = (duration % 24) as u64;
+        duration /= 24;
+        let years = (duration as f64 / 365.25) as u64;
+        let ydays = (duration as f64 % 365.25) as u64;
+        let months = (ydays as f64 / 30.44) as u64;
+        let mdays = (ydays as f64 % 30.44) as u64;
+        let weeks = mdays / 7;
+        let days = mdays % 7;
+        Self {
+            nsec,
+            msec,
+            usec,
+            seconds,
+            minutes,
+            hours,
+            days,
+            weeks,
+            months,
+            years,
+        }
+    }
+}
+
+impl std::fmt::Display for TimeSpan {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> Result<(), std::fmt::Error> {
+        let mut single = true;
+        if self.years > 0 {
+            write!(f, "{}y", self.years)?;
+            single = false;
+        }
+        let write_space = |single: bool, f: &mut std::fmt::Formatter| -> Result<bool, std::fmt::Error> {
+            if !single {
+                write!(f, " ")?;
+            }
+            Ok(false)
+        };
+        if self.months > 0 {
+            single = write_space(single, f)?;
+            write!(f, "{}m", self.months)?;
+        }
+        if self.weeks > 0 {
+            single = write_space(single, f)?;
+            write!(f, "{}w", self.weeks)?;
+        }
+        if self.days > 0 {
+            single = write_space(single, f)?;
+            write!(f, "{}d", self.days)?;
+        }
+        if self.hours > 0 {
+            single = write_space(single, f)?;
+            write!(f, "{}h", self.hours)?;
+        }
+        if self.minutes > 0 {
+            single = write_space(single, f)?;
+            write!(f, "{}min", self.minutes)?;
+        }
+        let seconds = self.seconds as f64 + (self.msec as f64 / 1000.0);
+        if seconds >= 0.1 {
+            write_space(single, f)?;
+            if seconds >= 1.0 || !single {
+                write!(f, "{:.0}s", seconds)?;
+            } else {
+                write!(f, "{:.1}s", seconds)?;
+            }
+        } else if single {
+            write_space(single, f)?;
+            write!(f, "<0.1s")?;
+        }
+        Ok(())
+    }
+}
 
 pub fn verify_time_span(i: &str) -> Result<(), Error> {
     parse_time_span(i)?;
-- 
2.20.1

[pbs-devel] [PATCH proxmox-backup 2/3] server/email_notifications: do not panic on template registration

[Dominik Csapak]

instead print an error and continue, the rendering functions will error
out if one of the templates could not be registered

if we `.unwrap()` here, it can lead to problems if the templates are
not correct, i.e. we could panic while holding a lock, if something holds
a mutex while this is called for the first time

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 src/server/email_notifications.rs | 35 +++++++++++++++++++------------
 1 file changed, 22 insertions(+), 13 deletions(-)

diff --git a/src/server/email_notifications.rs b/src/server/email_notifications.rs
index 229443f6..59cd016a 100644
--- a/src/server/email_notifications.rs
+++ b/src/server/email_notifications.rs
@@ -1,10 +1,11 @@
 use anyhow::Error;
 use serde_json::json;
 
-use handlebars::{Handlebars, Helper, Context, RenderError, RenderContext, Output, HelperResult};
+use handlebars::{Handlebars, Helper, Context, RenderError, RenderContext, Output, HelperResult, TemplateError};
 
 use proxmox::tools::email::sendmail;
 use proxmox::api::schema::parse_property_string;
+use proxmox::try_block;
 
 use crate::{
     config::datastore::DataStoreConfig,
@@ -181,25 +182,33 @@ lazy_static::lazy_static!{
 
     static ref HANDLEBARS: Handlebars<'static> = {
         let mut hb = Handlebars::new();
+        let result: Result<(), TemplateError> = try_block!({
 
-        hb.set_strict_mode(true);
+            hb.set_strict_mode(true);
 
-        hb.register_helper("human-bytes", Box::new(handlebars_humam_bytes_helper));
-        hb.register_helper("relative-percentage", Box::new(handlebars_relative_percentage_helper));
+            hb.register_helper("human-bytes", Box::new(handlebars_humam_bytes_helper));
+            hb.register_helper("relative-percentage", Box::new(handlebars_relative_percentage_helper));
 
-        hb.register_template_string("gc_ok_template", GC_OK_TEMPLATE).unwrap();
-        hb.register_template_string("gc_err_template", GC_ERR_TEMPLATE).unwrap();
+            hb.register_template_string("gc_ok_template", GC_OK_TEMPLATE)?;
+            hb.register_template_string("gc_err_template", GC_ERR_TEMPLATE)?;
 
-        hb.register_template_string("verify_ok_template", VERIFY_OK_TEMPLATE).unwrap();
-        hb.register_template_string("verify_err_template", VERIFY_ERR_TEMPLATE).unwrap();
+            hb.register_template_string("verify_ok_template", VERIFY_OK_TEMPLATE)?;
+            hb.register_template_string("verify_err_template", VERIFY_ERR_TEMPLATE)?;
 
-        hb.register_template_string("sync_ok_template", SYNC_OK_TEMPLATE).unwrap();
-        hb.register_template_string("sync_err_template", SYNC_ERR_TEMPLATE).unwrap();
+            hb.register_template_string("sync_ok_template", SYNC_OK_TEMPLATE)?;
+            hb.register_template_string("sync_err_template", SYNC_ERR_TEMPLATE)?;
 
-        hb.register_template_string("tape_backup_ok_template", TAPE_BACKUP_OK_TEMPLATE).unwrap();
-        hb.register_template_string("tape_backup_err_template", TAPE_BACKUP_ERR_TEMPLATE).unwrap();
+            hb.register_template_string("tape_backup_ok_template", TAPE_BACKUP_OK_TEMPLATE)?;
+            hb.register_template_string("tape_backup_err_template", TAPE_BACKUP_ERR_TEMPLATE)?;
 
-        hb.register_template_string("package_update_template", PACKAGE_UPDATES_TEMPLATE).unwrap();
+            hb.register_template_string("package_update_template", PACKAGE_UPDATES_TEMPLATE)?;
+
+            Ok(())
+        });
+
+        if let Err(err) = result {
+            eprintln!("error during template registration: {}", err);
+        }
 
         hb
     };
-- 
2.20.1

[pbs-devel] [PATCH proxmox-backup 3/3] api2/tape/backup: include a summary on notification e-mails

[Dominik Csapak]

for now only contains the list of included snapshots (if any),
as well as the backup duration

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 src/api2/tape/backup.rs           | 32 +++++++++++++++++++++++++------
 src/api2/types/tape/mod.rs        |  9 +++++++++
 src/server/email_notifications.rs | 13 +++++++++++++
 3 files changed, 48 insertions(+), 6 deletions(-)

diff --git a/src/api2/tape/backup.rs b/src/api2/tape/backup.rs
index 9f1167fe..5f7a16b4 100644
--- a/src/api2/tape/backup.rs
+++ b/src/api2/tape/backup.rs
@@ -51,6 +51,7 @@ use crate::{
         JOB_ID_SCHEMA,
         MediaPoolConfig,
         Userid,
+        TapeBackupJobSummary,
     },
     server::WorkerTask,
     task::TaskState,
@@ -198,13 +199,16 @@ pub fn do_tape_backup_job(
             let notify_user = setup.notify_user.as_ref().unwrap_or_else(|| &Userid::root_userid());
             let email = lookup_user_email(notify_user);
 
-            let job_result = backup_worker(
+            let (job_result, summary) = match backup_worker(
                 &worker,
                 datastore,
                 &pool_config,
                 &setup,
                 email.clone(),
-            );
+            ) {
+                Ok(summary) => (Ok(()), summary),
+                Err(err) => (Err(err), Default::default()),
+            };
 
             let status = worker.create_state(&job_result);
 
@@ -214,6 +218,7 @@ pub fn do_tape_backup_job(
                     Some(job.jobname()),
                     &setup,
                     &job_result,
+                    summary,
                 ) {
                     eprintln!("send tape backup notification failed: {}", err);
                 }
@@ -340,13 +345,17 @@ pub fn backup(
         move |worker| {
             let _drive_lock = drive_lock; // keep lock guard
             set_tape_device_state(&setup.drive, &worker.upid().to_string())?;
-            let job_result = backup_worker(
+
+            let (job_result, summary) = match backup_worker(
                 &worker,
                 datastore,
                 &pool_config,
                 &setup,
                 email.clone(),
-            );
+            ) {
+                Ok(summary) => (Ok(()), summary),
+                Err(err) => (Err(err), Default::default()),
+            };
 
             if let Some(email) = email {
                 if let Err(err) = crate::server::send_tape_backup_status(
@@ -354,6 +363,7 @@ pub fn backup(
                     None,
                     &setup,
                     &job_result,
+                    summary,
                 ) {
                     eprintln!("send tape backup notification failed: {}", err);
                 }
@@ -374,9 +384,11 @@ fn backup_worker(
     pool_config: &MediaPoolConfig,
     setup: &TapeBackupJobSetup,
     email: Option<String>,
-) -> Result<(), Error> {
+) -> Result<TapeBackupJobSummary, Error> {
 
     let status_path = Path::new(TAPE_STATUS_DIR);
+    let start = std::time::Instant::now();
+    let mut summary: TapeBackupJobSummary = Default::default();
 
     let _lock = MediaPool::lock(status_path, &pool_config.name)?;
 
@@ -420,8 +432,11 @@ fn backup_worker(
                     task_log!(worker, "skip snapshot {}", info.backup_dir);
                     continue;
                 }
+                let snapshot_name = info.backup_dir.to_string();
                 if !backup_snapshot(worker, &mut pool_writer, datastore.clone(), info.backup_dir)? {
                     errors = true;
+                } else {
+                    summary.snapshot_list.push(snapshot_name);
                 }
                 progress.done_snapshots = 1;
                 task_log!(
@@ -437,8 +452,11 @@ fn backup_worker(
                     task_log!(worker, "skip snapshot {}", info.backup_dir);
                     continue;
                 }
+                let snapshot_name = info.backup_dir.to_string();
                 if !backup_snapshot(worker, &mut pool_writer, datastore.clone(), info.backup_dir)? {
                     errors = true;
+                } else {
+                    summary.snapshot_list.push(snapshot_name);
                 }
                 progress.done_snapshots = snapshot_number as u64 + 1;
                 task_log!(
@@ -462,7 +480,9 @@ fn backup_worker(
         bail!("Tape backup finished with some errors. Please check the task log.");
     }
 
-    Ok(())
+    summary.duration = start.elapsed();
+
+    Ok(summary)
 }
 
 // Try to update the the media online status
diff --git a/src/api2/types/tape/mod.rs b/src/api2/types/tape/mod.rs
index 68b2cf12..44c2236b 100644
--- a/src/api2/types/tape/mod.rs
+++ b/src/api2/types/tape/mod.rs
@@ -20,3 +20,12 @@ pub use media_location::*;
 
 mod media;
 pub use media::*;
+
+/// Summary of a successful Tape Job
+#[derive(Default)]
+pub struct TapeBackupJobSummary {
+    /// The list of snaphots backed up
+    pub snapshot_list: Vec<String>,
+    /// The total time of the backup job
+    pub duration: std::time::Duration,
+}
diff --git a/src/server/email_notifications.rs b/src/server/email_notifications.rs
index 59cd016a..70e13053 100644
--- a/src/server/email_notifications.rs
+++ b/src/server/email_notifications.rs
@@ -18,6 +18,7 @@ use crate::{
         Userid,
         Notify,
         DatastoreNotify,
+        TapeBackupJobSummary,
     },
     tools::format::HumanByte,
 };
@@ -149,6 +150,14 @@ Datastore:  {{job.store}}
 Tape Pool:  {{job.pool}}
 Tape Drive: {{job.drive}}
 
+{{#if snapshot-list ~}}
+Snapshots included:
+
+{{#each snapshot-list~}}
+{{this}}
+{{/each~}}
+{{/if}}
+Duration: {{duration}}
 
 Tape Backup successful.
 
@@ -411,14 +420,18 @@ pub fn send_tape_backup_status(
     id: Option<&str>,
     job: &TapeBackupJobSetup,
     result: &Result<(), Error>,
+    summary: TapeBackupJobSummary,
 ) -> Result<(), Error> {
 
     let (fqdn, port) = get_server_url();
+    let duration: crate::tools::systemd::time::TimeSpan = summary.duration.into();
     let mut data = json!({
         "job": job,
         "fqdn": fqdn,
         "port": port,
         "id": id,
+        "snapshot-list": summary.snapshot_list,
+        "duration": duration.to_string(),
     });
 
     let text = match result {
-- 
2.20.1

Re: [pbs-devel] [PATCH proxmox-backup 1/3] tools/systemd/time: implement some Traits for TimeSpan

[Thomas Lamprecht]

On 16.03.21 12:56, Dominik Csapak wrote:
> namely
> * From<Duration> (to convert easily from duration to timespan)
> * Display (for better formatting)
> 
> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
> ---
> if wanted, we can optimize the display trait a bit further, e.g.
> only showing the biggest two units instead
> 
>  src/tools/systemd/time.rs | 84 +++++++++++++++++++++++++++++++++++++++
>  1 file changed, 84 insertions(+)
> 
> diff --git a/src/tools/systemd/time.rs b/src/tools/systemd/time.rs
> index 7cc42415..75fb0ea2 100644
> --- a/src/tools/systemd/time.rs
> +++ b/src/tools/systemd/time.rs
> @@ -141,6 +141,90 @@ impl From<TimeSpan> for f64 {
>      }
>  }
>  
> +impl From<std::time::Duration> for TimeSpan {
> +    fn from(duration: std::time::Duration) -> Self {
> +        let mut duration = duration.as_nanos();
> +        let nsec = (duration % 1000) as u64;
> +        duration /= 1000;
> +        let usec = (duration % 1000) as u64;
> +        duration /= 1000;
> +        let msec = (duration % 1000) as u64;
> +        duration /= 1000;
> +        let seconds = (duration % 60) as u64;
> +        duration /= 60;
> +        let minutes = (duration % 60) as u64;
> +        duration /= 60;
> +        let hours = (duration % 24) as u64;
> +        duration /= 24;
> +        let years = (duration as f64 / 365.25) as u64;
> +        let ydays = (duration as f64 % 365.25) as u64;
> +        let months = (ydays as f64 / 30.44) as u64;
> +        let mdays = (ydays as f64 % 30.44) as u64;
> +        let weeks = mdays / 7;
> +        let days = mdays % 7;
> +        Self {
> +            nsec,
> +            msec,
> +            usec,

nit:
nano -> milli -> micro? While it works correct, that order above seems just wrong...

> +            seconds,
> +            minutes,
> +            hours,
> +            days,
> +            weeks,
> +            months,
> +            years,
> +        }
> +    }
> +}
> +
> +impl std::fmt::Display for TimeSpan {
> +    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> Result<(), std::fmt::Error> {
> +        let mut single = true;
> +        if self.years > 0 {
> +            write!(f, "{}y", self.years)?;
> +            single = false;
> +        }
> +        let write_space = |single: bool, f: &mut std::fmt::Formatter| -> Result<bool, std::fmt::Error> {
> +            if !single {
> +                write!(f, " ")?;
> +            }
> +            Ok(false)
> +        };

No sure if something like (untested):

let mut first = true;
let mut do_write = |v: u64, unit: &str| -> Result<(), std::fmt::Error> {
    if !first {
        write!(f, " ")?;
        first = false;
    }
    write!(f, "{}{}", v, unit)
}

if self.months > 0 {
    do_write(self.months, "m")?;
}

...

would be nicer IMO, especially as the "single" handling reads a bit weird.
Maybe the seconds field makes this not really feasible though...

> +        if self.months > 0 {
> +            single = write_space(single, f)?;
> +            write!(f, "{}m", self.months)?;
> +        }
> +        if self.weeks > 0 {
> +            single = write_space(single, f)?;
> +            write!(f, "{}w", self.weeks)?;
> +        }
> +        if self.days > 0 {
> +            single = write_space(single, f)?;
> +            write!(f, "{}d", self.days)?;
> +        }
> +        if self.hours > 0 {
> +            single = write_space(single, f)?;
> +            write!(f, "{}h", self.hours)?;
> +        }
> +        if self.minutes > 0 {
> +            single = write_space(single, f)?;
> +            write!(f, "{}min", self.minutes)?;
> +        }
> +        let seconds = self.seconds as f64 + (self.msec as f64 / 1000.0);
> +        if seconds >= 0.1 {
> +            write_space(single, f)?;
> +            if seconds >= 1.0 || !single {
> +                write!(f, "{:.0}s", seconds)?;
> +            } else {
> +                write!(f, "{:.1}s", seconds)?;
> +            }
> +        } else if single {
> +            write_space(single, f)?;
> +            write!(f, "<0.1s")?;
> +        }
> +        Ok(())
> +    }
> +}
>  
>  pub fn verify_time_span(i: &str) -> Result<(), Error> {
>      parse_time_span(i)?;
> 

Re: [pbs-devel] [PATCH proxmox-backup 2/3] server/email_notifications: do not panic on template registration

[Thomas Lamprecht]

On 16.03.21 12:56, Dominik Csapak wrote:
> instead print an error and continue, the rendering functions will error
> out if one of the templates could not be registered
> 
> if we `.unwrap()` here, it can lead to problems if the templates are
> not correct, i.e. we could panic while holding a lock, if something holds
> a mutex while this is called for the first time

how can they error?
And any error (with or without this patch) would lead to emails notification not
working anymore, some may seem this as quite fatal error if they do not get notified
on erroneous jobs anymore? We may not be able to do much here, that's why above
question about what the error source can be.

> 
> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
> ---
>  src/server/email_notifications.rs | 35 +++++++++++++++++++------------
>  1 file changed, 22 insertions(+), 13 deletions(-)
> 
> diff --git a/src/server/email_notifications.rs b/src/server/email_notifications.rs
> index 229443f6..59cd016a 100644
> --- a/src/server/email_notifications.rs
> +++ b/src/server/email_notifications.rs
> @@ -1,10 +1,11 @@
>  use anyhow::Error;
>  use serde_json::json;
>  
> -use handlebars::{Handlebars, Helper, Context, RenderError, RenderContext, Output, HelperResult};
> +use handlebars::{Handlebars, Helper, Context, RenderError, RenderContext, Output, HelperResult, TemplateError};
>  
>  use proxmox::tools::email::sendmail;
>  use proxmox::api::schema::parse_property_string;
> +use proxmox::try_block;
>  
>  use crate::{
>      config::datastore::DataStoreConfig,
> @@ -181,25 +182,33 @@ lazy_static::lazy_static!{
>  
>      static ref HANDLEBARS: Handlebars<'static> = {
>          let mut hb = Handlebars::new();
> +        let result: Result<(), TemplateError> = try_block!({
>  
> -        hb.set_strict_mode(true);
> +            hb.set_strict_mode(true);
>  
> -        hb.register_helper("human-bytes", Box::new(handlebars_humam_bytes_helper));
> -        hb.register_helper("relative-percentage", Box::new(handlebars_relative_percentage_helper));
> +            hb.register_helper("human-bytes", Box::new(handlebars_humam_bytes_helper));
> +            hb.register_helper("relative-percentage", Box::new(handlebars_relative_percentage_helper));
>  
> -        hb.register_template_string("gc_ok_template", GC_OK_TEMPLATE).unwrap();
> -        hb.register_template_string("gc_err_template", GC_ERR_TEMPLATE).unwrap();
> +            hb.register_template_string("gc_ok_template", GC_OK_TEMPLATE)?;
> +            hb.register_template_string("gc_err_template", GC_ERR_TEMPLATE)?;
>  
> -        hb.register_template_string("verify_ok_template", VERIFY_OK_TEMPLATE).unwrap();
> -        hb.register_template_string("verify_err_template", VERIFY_ERR_TEMPLATE).unwrap();
> +            hb.register_template_string("verify_ok_template", VERIFY_OK_TEMPLATE)?;
> +            hb.register_template_string("verify_err_template", VERIFY_ERR_TEMPLATE)?;
>  
> -        hb.register_template_string("sync_ok_template", SYNC_OK_TEMPLATE).unwrap();
> -        hb.register_template_string("sync_err_template", SYNC_ERR_TEMPLATE).unwrap();
> +            hb.register_template_string("sync_ok_template", SYNC_OK_TEMPLATE)?;
> +            hb.register_template_string("sync_err_template", SYNC_ERR_TEMPLATE)?;
>  
> -        hb.register_template_string("tape_backup_ok_template", TAPE_BACKUP_OK_TEMPLATE).unwrap();
> -        hb.register_template_string("tape_backup_err_template", TAPE_BACKUP_ERR_TEMPLATE).unwrap();
> +            hb.register_template_string("tape_backup_ok_template", TAPE_BACKUP_OK_TEMPLATE)?;
> +            hb.register_template_string("tape_backup_err_template", TAPE_BACKUP_ERR_TEMPLATE)?;
>  
> -        hb.register_template_string("package_update_template", PACKAGE_UPDATES_TEMPLATE).unwrap();
> +            hb.register_template_string("package_update_template", PACKAGE_UPDATES_TEMPLATE)?;
> +
> +            Ok(())
> +        });
> +
> +        if let Err(err) = result {
> +            eprintln!("error during template registration: {}", err);
> +        }
>  
>          hb
>      };
> 

Re: [pbs-devel] [PATCH proxmox-backup 1/3] tools/systemd/time: implement some Traits for TimeSpan

[Dominik Csapak]

On 3/17/21 20:35, Thomas Lamprecht wrote:
> On 16.03.21 12:56, Dominik Csapak wrote:
>> namely
>> * From<Duration> (to convert easily from duration to timespan)
>> * Display (for better formatting)
>>
>> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
>> ---
>> if wanted, we can optimize the display trait a bit further, e.g.
>> only showing the biggest two units instead
>>
>>   src/tools/systemd/time.rs | 84 +++++++++++++++++++++++++++++++++++++++
>>   1 file changed, 84 insertions(+)
>>
>> diff --git a/src/tools/systemd/time.rs b/src/tools/systemd/time.rs
>> index 7cc42415..75fb0ea2 100644
>> --- a/src/tools/systemd/time.rs
>> +++ b/src/tools/systemd/time.rs
>> @@ -141,6 +141,90 @@ impl From<TimeSpan> for f64 {
>>       }
>>   }
>>   
>> +impl From<std::time::Duration> for TimeSpan {
>> +    fn from(duration: std::time::Duration) -> Self {
>> +        let mut duration = duration.as_nanos();
>> +        let nsec = (duration % 1000) as u64;
>> +        duration /= 1000;
>> +        let usec = (duration % 1000) as u64;
>> +        duration /= 1000;
>> +        let msec = (duration % 1000) as u64;
>> +        duration /= 1000;
>> +        let seconds = (duration % 60) as u64;
>> +        duration /= 60;
>> +        let minutes = (duration % 60) as u64;
>> +        duration /= 60;
>> +        let hours = (duration % 24) as u64;
>> +        duration /= 24;
>> +        let years = (duration as f64 / 365.25) as u64;
>> +        let ydays = (duration as f64 % 365.25) as u64;
>> +        let months = (ydays as f64 / 30.44) as u64;
>> +        let mdays = (ydays as f64 % 30.44) as u64;
>> +        let weeks = mdays / 7;
>> +        let days = mdays % 7;
>> +        Self {
>> +            nsec,
>> +            msec,
>> +            usec,
> 
> nit:
> nano -> milli -> micro? While it works correct, that order above seems just wrong...

yeah sorry...


> 
>> +            seconds,
>> +            minutes,
>> +            hours,
>> +            days,
>> +            weeks,
>> +            months,
>> +            years,
>> +        }
>> +    }
>> +}
>> +
>> +impl std::fmt::Display for TimeSpan {
>> +    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> Result<(), std::fmt::Error> {
>> +        let mut single = true;
>> +        if self.years > 0 {
>> +            write!(f, "{}y", self.years)?;
>> +            single = false;
>> +        }
>> +        let write_space = |single: bool, f: &mut std::fmt::Formatter| -> Result<bool, std::fmt::Error> {
>> +            if !single {
>> +                write!(f, " ")?;
>> +            }
>> +            Ok(false)
>> +        };
> 
> No sure if something like (untested):
> 
> let mut first = true;
> let mut do_write = |v: u64, unit: &str| -> Result<(), std::fmt::Error> {
>      if !first {
>          write!(f, " ")?;
>          first = false;
>      }
>      write!(f, "{}{}", v, unit)
> }
> 
> if self.months > 0 {
>      do_write(self.months, "m")?;
> }
> 
> ...
> 
> would be nicer IMO, especially as the "single" handling reads a bit weird.
> Maybe the seconds field makes this not really feasible though...

mhm.. i can try to improve that a bit

> 
>> +        if self.months > 0 {
>> +            single = write_space(single, f)?;
>> +            write!(f, "{}m", self.months)?;
>> +        }
>> +        if self.weeks > 0 {
>> +            single = write_space(single, f)?;
>> +            write!(f, "{}w", self.weeks)?;
>> +        }
>> +        if self.days > 0 {
>> +            single = write_space(single, f)?;
>> +            write!(f, "{}d", self.days)?;
>> +        }
>> +        if self.hours > 0 {
>> +            single = write_space(single, f)?;
>> +            write!(f, "{}h", self.hours)?;
>> +        }
>> +        if self.minutes > 0 {
>> +            single = write_space(single, f)?;
>> +            write!(f, "{}min", self.minutes)?;
>> +        }
>> +        let seconds = self.seconds as f64 + (self.msec as f64 / 1000.0);
>> +        if seconds >= 0.1 {
>> +            write_space(single, f)?;
>> +            if seconds >= 1.0 || !single {
>> +                write!(f, "{:.0}s", seconds)?;
>> +            } else {
>> +                write!(f, "{:.1}s", seconds)?;
>> +            }
>> +        } else if single {
>> +            write_space(single, f)?;
>> +            write!(f, "<0.1s")?;
>> +        }
>> +        Ok(())
>> +    }
>> +}
>>   
>>   pub fn verify_time_span(i: &str) -> Result<(), Error> {
>>       parse_time_span(i)?;
>>
> 

Re: [pbs-devel] [PATCH proxmox-backup 2/3] server/email_notifications: do not panic on template registration

[Dominik Csapak]

On 3/17/21 20:38, Thomas Lamprecht wrote:
> On 16.03.21 12:56, Dominik Csapak wrote:
>> instead print an error and continue, the rendering functions will error
>> out if one of the templates could not be registered
>>
>> if we `.unwrap()` here, it can lead to problems if the templates are
>> not correct, i.e. we could panic while holding a lock, if something holds
>> a mutex while this is called for the first time
> 
> how can they error?
> And any error (with or without this patch) would lead to emails notification not
> working anymore, some may seem this as quite fatal error if they do not get notified
> on erroneous jobs anymore? We may not be able to do much here, that's why above
> question about what the error source can be.

they can error if they do not compile, e.g. they have syntax errors
while we should catch that during developement/reviewing,
if it does happen, it generates some weird behaviour
(for example panicing while holding a mutex)

with my patch, we still generate a warning, but aside from
sending notification mails (where we still would warn in the log)
the rest should work fine, there we can ofc also error out on
notification errors so that the tasks get an error
(but a well defined one instead of a panic)

also we may want to put the templates into files in the future
so that users can adapt it and we can more easily change
them (maybe localize them?)

> 
>>
>> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
>> ---
>>   src/server/email_notifications.rs | 35 +++++++++++++++++++------------
>>   1 file changed, 22 insertions(+), 13 deletions(-)
>>
>> diff --git a/src/server/email_notifications.rs b/src/server/email_notifications.rs
>> index 229443f6..59cd016a 100644
>> --- a/src/server/email_notifications.rs
>> +++ b/src/server/email_notifications.rs
>> @@ -1,10 +1,11 @@
>>   use anyhow::Error;
>>   use serde_json::json;
>>   
>> -use handlebars::{Handlebars, Helper, Context, RenderError, RenderContext, Output, HelperResult};
>> +use handlebars::{Handlebars, Helper, Context, RenderError, RenderContext, Output, HelperResult, TemplateError};
>>   
>>   use proxmox::tools::email::sendmail;
>>   use proxmox::api::schema::parse_property_string;
>> +use proxmox::try_block;
>>   
>>   use crate::{
>>       config::datastore::DataStoreConfig,
>> @@ -181,25 +182,33 @@ lazy_static::lazy_static!{
>>   
>>       static ref HANDLEBARS: Handlebars<'static> = {
>>           let mut hb = Handlebars::new();
>> +        let result: Result<(), TemplateError> = try_block!({
>>   
>> -        hb.set_strict_mode(true);
>> +            hb.set_strict_mode(true);
>>   
>> -        hb.register_helper("human-bytes", Box::new(handlebars_humam_bytes_helper));
>> -        hb.register_helper("relative-percentage", Box::new(handlebars_relative_percentage_helper));
>> +            hb.register_helper("human-bytes", Box::new(handlebars_humam_bytes_helper));
>> +            hb.register_helper("relative-percentage", Box::new(handlebars_relative_percentage_helper));
>>   
>> -        hb.register_template_string("gc_ok_template", GC_OK_TEMPLATE).unwrap();
>> -        hb.register_template_string("gc_err_template", GC_ERR_TEMPLATE).unwrap();
>> +            hb.register_template_string("gc_ok_template", GC_OK_TEMPLATE)?;
>> +            hb.register_template_string("gc_err_template", GC_ERR_TEMPLATE)?;
>>   
>> -        hb.register_template_string("verify_ok_template", VERIFY_OK_TEMPLATE).unwrap();
>> -        hb.register_template_string("verify_err_template", VERIFY_ERR_TEMPLATE).unwrap();
>> +            hb.register_template_string("verify_ok_template", VERIFY_OK_TEMPLATE)?;
>> +            hb.register_template_string("verify_err_template", VERIFY_ERR_TEMPLATE)?;
>>   
>> -        hb.register_template_string("sync_ok_template", SYNC_OK_TEMPLATE).unwrap();
>> -        hb.register_template_string("sync_err_template", SYNC_ERR_TEMPLATE).unwrap();
>> +            hb.register_template_string("sync_ok_template", SYNC_OK_TEMPLATE)?;
>> +            hb.register_template_string("sync_err_template", SYNC_ERR_TEMPLATE)?;
>>   
>> -        hb.register_template_string("tape_backup_ok_template", TAPE_BACKUP_OK_TEMPLATE).unwrap();
>> -        hb.register_template_string("tape_backup_err_template", TAPE_BACKUP_ERR_TEMPLATE).unwrap();
>> +            hb.register_template_string("tape_backup_ok_template", TAPE_BACKUP_OK_TEMPLATE)?;
>> +            hb.register_template_string("tape_backup_err_template", TAPE_BACKUP_ERR_TEMPLATE)?;
>>   
>> -        hb.register_template_string("package_update_template", PACKAGE_UPDATES_TEMPLATE).unwrap();
>> +            hb.register_template_string("package_update_template", PACKAGE_UPDATES_TEMPLATE)?;
>> +
>> +            Ok(())
>> +        });
>> +
>> +        if let Err(err) = result {
>> +            eprintln!("error during template registration: {}", err);
>> +        }
>>   
>>           hb
>>       };
>>
> 

Re: [pbs-devel] [PATCH proxmox-backup 2/3] server/email_notifications: do not panic on template registration

[Thomas Lamprecht]

On 18.03.21 10:57, Dominik Csapak wrote:
> On 3/17/21 20:38, Thomas Lamprecht wrote:
>> On 16.03.21 12:56, Dominik Csapak wrote:
>>> instead print an error and continue, the rendering functions will error
>>> out if one of the templates could not be registered
>>>
>>> if we `.unwrap()` here, it can lead to problems if the templates are
>>> not correct, i.e. we could panic while holding a lock, if something holds
>>> a mutex while this is called for the first time
>>
>> how can they error?
>> And any error (with or without this patch) would lead to emails notification not
>> working anymore, some may seem this as quite fatal error if they do not get notified
>> on erroneous jobs anymore? We may not be able to do much here, that's why above
>> question about what the error source can be.
> 
> they can error if they do not compile, e.g. they have syntax errors
> while we should catch that during developement/reviewing,
> if it does happen, it generates some weird behaviour
> (for example panicing while holding a mutex)

can't we just add a test for that instead, so that it is actually "compile checked"
when building a package?

Then such errors would be actually fixed before getting released, relying on
review/test tends to fail and let slip something trhough sooner or later.
> 
> with my patch, we still generate a warning, but aside from
> sending notification mails (where we still would warn in the log)
> the rest should work fine, there we can ofc also error out on
> notification errors so that the tasks get an error
> (but a well defined one instead of a panic)
> 
> also we may want to put the templates into files in the future
> so that users can adapt it and we can more easily change
> them (maybe localize them?)
> 

would need a sane reload mechanism then though, and in any way the one we ship
should be tested for basic validity on build.

Re: [pbs-devel] [PATCH proxmox-backup 2/3] server/email_notifications: do not panic on template registration

[Dominik Csapak]

On 3/18/21 11:21, Thomas Lamprecht wrote:
> On 18.03.21 10:57, Dominik Csapak wrote:
>> On 3/17/21 20:38, Thomas Lamprecht wrote:
>>> On 16.03.21 12:56, Dominik Csapak wrote:
>>>> instead print an error and continue, the rendering functions will error
>>>> out if one of the templates could not be registered
>>>>
>>>> if we `.unwrap()` here, it can lead to problems if the templates are
>>>> not correct, i.e. we could panic while holding a lock, if something holds
>>>> a mutex while this is called for the first time
>>>
>>> how can they error?
>>> And any error (with or without this patch) would lead to emails notification not
>>> working anymore, some may seem this as quite fatal error if they do not get notified
>>> on erroneous jobs anymore? We may not be able to do much here, that's why above
>>> question about what the error source can be.
>>
>> they can error if they do not compile, e.g. they have syntax errors
>> while we should catch that during developement/reviewing,
>> if it does happen, it generates some weird behaviour
>> (for example panicing while holding a mutex)
> 
> can't we just add a test for that instead, so that it is actually "compile checked"
> when building a package?
> 
> Then such errors would be actually fixed before getting released, relying on > review/test tends to fail and let slip something trhough sooner or later.

yes ofc, thats a good idea, nonetheless would i prefer to not panic in 
such a situation, especially if there are no real downsides

>>
>> with my patch, we still generate a warning, but aside from
>> sending notification mails (where we still would warn in the log)
>> the rest should work fine, there we can ofc also error out on
>> notification errors so that the tasks get an error
>> (but a well defined one instead of a panic)
>>
>> also we may want to put the templates into files in the future
>> so that users can adapt it and we can more easily change
>> them (maybe localize them?)
>>
> 
> would need a sane reload mechanism then though, and in any way the one we ship
> should be tested for basic validity on build.
> 

wouldn't it be enough to leave it like it is and document the
necessary daemon reload for use customized templates?

(i'd imagine a system like for pmg, where we have the
shipped templates separate from the ones the user
customizes)

on package update we reload the daemon anyway, so it reloads
the new templates

also we could ship a 'check-template' binary if we really
wanted to

Re: [pbs-devel] [PATCH proxmox-backup 2/3] server/email_notifications: do not panic on template registration

[Thomas Lamprecht]

On 18.03.21 11:31, Dominik Csapak wrote:
> On 3/18/21 11:21, Thomas Lamprecht wrote:
>> On 18.03.21 10:57, Dominik Csapak wrote:
>>> On 3/17/21 20:38, Thomas Lamprecht wrote:
>>>> On 16.03.21 12:56, Dominik Csapak wrote:
>>>>> instead print an error and continue, the rendering functions will error
>>>>> out if one of the templates could not be registered
>>>>>
>>>>> if we `.unwrap()` here, it can lead to problems if the templates are
>>>>> not correct, i.e. we could panic while holding a lock, if something holds
>>>>> a mutex while this is called for the first time
>>>>
>>>> how can they error?
>>>> And any error (with or without this patch) would lead to emails notification not
>>>> working anymore, some may seem this as quite fatal error if they do not get notified
>>>> on erroneous jobs anymore? We may not be able to do much here, that's why above
>>>> question about what the error source can be.
>>>
>>> they can error if they do not compile, e.g. they have syntax errors
>>> while we should catch that during developement/reviewing,
>>> if it does happen, it generates some weird behaviour
>>> (for example panicing while holding a mutex)
>>
>> can't we just add a test for that instead, so that it is actually "compile checked"
>> when building a package?
>>
>> Then such errors would be actually fixed before getting released, relying on > review/test tends to fail and let slip something trhough sooner or later.
> 
> yes ofc, thats a good idea, nonetheless would i prefer to not panic in such a situation, especially if there are no real downsides

I mean if those situation cannot happen in practice anymore I see no
reason for band-aiding a theoretical problem...
But if you really want, OK, but I'd prefer to only take it in with the testing
in place; to avoid that such errors are now completely missed as overlooking
a log message is easy.

>>> also we may want to put the templates into files in the future
>>> so that users can adapt it and we can more easily change
>>> them (maybe localize them?)
>>>
>>
>> would need a sane reload mechanism then though, and in any way the one we ship
>> should be tested for basic validity on build.
>>
> 
> wouldn't it be enough to leave it like it is and document the
> necessary daemon reload for use customized templates?

reload isn't exactly cheap, and we want to avoid that whenever possible, that
includes things like template reloads or certificates update (in the future).

> 
> (i'd imagine a system like for pmg, where we have the
> shipped templates separate from the ones the user
> customizes)

I'd wait for file templates until some are actually requested for a sensible
use case..