all lists on lists.proxmox.com
 help / color / mirror / Atom feed
* [pbs-devel] [PATCH proxmox-backup v2 0/5] improve tape backup and notifications
@ 2021-03-18 12:01 Dominik Csapak
  2021-03-18 12:01 ` [pbs-devel] [PATCH proxmox-backup v2 1/5] tools/systemd/time: implement some Traits for TimeSpan Dominik Csapak
                   ` (5 more replies)
  0 siblings, 6 replies; 8+ messages in thread
From: Dominik Csapak @ 2021-03-18 12:01 UTC (permalink / raw)
  To: pbs-devel

combines my last three series[0][1][2], since they touch the same code

changes from v1:
* improve display trait code for timespan
* add tests for template registration
* rebase on master

0: https://lists.proxmox.com/pipermail/pbs-devel/2021-March/002438.html
1: https://lists.proxmox.com/pipermail/pbs-devel/2021-March/002441.html
2: https://lists.proxmox.com/pipermail/pbs-devel/2021-March/002431.html

Dominik Csapak (5):
  tools/systemd/time: implement some Traits for TimeSpan
  server/email_notifications: do not panic on template registration
  server/email_notifications: do not double html escape
  api2/tape/backup: include a summary on notification e-mails
  api2/tape/backup: wait indefinitely for lock in scheduled backup jobs

 src/api2/tape/backup.rs           | 85 ++++++++++++++++++++++---------
 src/api2/types/tape/mod.rs        |  9 ++++
 src/server/email_notifications.rs | 69 ++++++++++++++++++++-----
 src/tools/systemd/time.rs         | 82 +++++++++++++++++++++++++++++
 4 files changed, 209 insertions(+), 36 deletions(-)

-- 
2.20.1





^ permalink raw reply	[flat|nested] 8+ messages in thread

* [pbs-devel] [PATCH proxmox-backup v2 1/5] tools/systemd/time: implement some Traits for TimeSpan
  2021-03-18 12:01 [pbs-devel] [PATCH proxmox-backup v2 0/5] improve tape backup and notifications Dominik Csapak
@ 2021-03-18 12:01 ` Dominik Csapak
  2021-03-18 12:01 ` [pbs-devel] [PATCH proxmox-backup v2 2/5] server/email_notifications: do not panic on template registration Dominik Csapak
                   ` (4 subsequent siblings)
  5 siblings, 0 replies; 8+ messages in thread
From: Dominik Csapak @ 2021-03-18 12:01 UTC (permalink / raw)
  To: pbs-devel

namely
* From<Duration> (to convert easily from duration to timespan)
* Display (for better formatting)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
changes from v1:
* reorder msec, usec
* improve display trait code

 src/tools/systemd/time.rs | 82 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 82 insertions(+)

diff --git a/src/tools/systemd/time.rs b/src/tools/systemd/time.rs
index 7cc42415..cbcb23df 100644
--- a/src/tools/systemd/time.rs
+++ b/src/tools/systemd/time.rs
@@ -141,6 +141,88 @@ impl From<TimeSpan> for f64 {
     }
 }
 
+impl From<std::time::Duration> for TimeSpan {
+    fn from(duration: std::time::Duration) -> Self {
+        let mut duration = duration.as_nanos();
+        let nsec = (duration % 1000) as u64;
+        duration /= 1000;
+        let usec = (duration % 1000) as u64;
+        duration /= 1000;
+        let msec = (duration % 1000) as u64;
+        duration /= 1000;
+        let seconds = (duration % 60) as u64;
+        duration /= 60;
+        let minutes = (duration % 60) as u64;
+        duration /= 60;
+        let hours = (duration % 24) as u64;
+        duration /= 24;
+        let years = (duration as f64 / 365.25) as u64;
+        let ydays = (duration as f64 % 365.25) as u64;
+        let months = (ydays as f64 / 30.44) as u64;
+        let mdays = (ydays as f64 % 30.44) as u64;
+        let weeks = mdays / 7;
+        let days = mdays % 7;
+        Self {
+            nsec,
+            usec,
+            msec,
+            seconds,
+            minutes,
+            hours,
+            days,
+            weeks,
+            months,
+            years,
+        }
+    }
+}
+
+impl std::fmt::Display for TimeSpan {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> Result<(), std::fmt::Error> {
+        let mut first = true;
+        { // block scope for mutable borrows
+            let mut do_write = |v: u64, unit: &str| -> Result<(), std::fmt::Error> {
+                if !first {
+                    write!(f, " ")?;
+                }
+                first = false;
+                write!(f, "{}{}", v, unit)
+            };
+            if self.years > 0 {
+                do_write(self.years, "y")?;
+            }
+            if self.months > 0 {
+                do_write(self.months, "m")?;
+            }
+            if self.weeks > 0 {
+                do_write(self.weeks, "w")?;
+            }
+            if self.days > 0 {
+                do_write(self.days, "d")?;
+            }
+            if self.hours > 0 {
+                do_write(self.hours, "h")?;
+            }
+            if self.minutes > 0 {
+                do_write(self.minutes, "min")?;
+            }
+        }
+        if !first {
+            write!(f, " ")?;
+        }
+        let seconds = self.seconds as f64 + (self.msec as f64 / 1000.0);
+        if seconds >= 0.1 {
+            if seconds >= 1.0 || !first {
+                write!(f, "{:.0}s", seconds)?;
+            } else {
+                write!(f, "{:.1}s", seconds)?;
+            }
+        } else if first {
+            write!(f, "<0.1s")?;
+        }
+        Ok(())
+    }
+}
 
 pub fn verify_time_span(i: &str) -> Result<(), Error> {
     parse_time_span(i)?;
-- 
2.20.1





^ permalink raw reply	[flat|nested] 8+ messages in thread

* [pbs-devel] [PATCH proxmox-backup v2 2/5] server/email_notifications: do not panic on template registration
  2021-03-18 12:01 [pbs-devel] [PATCH proxmox-backup v2 0/5] improve tape backup and notifications Dominik Csapak
  2021-03-18 12:01 ` [pbs-devel] [PATCH proxmox-backup v2 1/5] tools/systemd/time: implement some Traits for TimeSpan Dominik Csapak
@ 2021-03-18 12:01 ` Dominik Csapak
  2021-03-18 12:01 ` [pbs-devel] [PATCH proxmox-backup v2 3/5] server/email_notifications: do not double html escape Dominik Csapak
                   ` (3 subsequent siblings)
  5 siblings, 0 replies; 8+ messages in thread
From: Dominik Csapak @ 2021-03-18 12:01 UTC (permalink / raw)
  To: pbs-devel

instead print an error and continue, the rendering functions will error
out if one of the templates could not be registered

if we `.unwrap()` here, it can lead to problems if the templates are
not correct, i.e. we could panic while holding a lock, if something holds
a mutex while this is called for the first time

add a test to catch registration issues during package build

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
changes from v1:
* add test to actually test registering, this will fail
  on package build if the template registering fails

 src/server/email_notifications.rs | 55 +++++++++++++++++++++++--------
 1 file changed, 42 insertions(+), 13 deletions(-)

diff --git a/src/server/email_notifications.rs b/src/server/email_notifications.rs
index 229443f6..cd5a43fe 100644
--- a/src/server/email_notifications.rs
+++ b/src/server/email_notifications.rs
@@ -1,10 +1,11 @@
 use anyhow::Error;
 use serde_json::json;
 
-use handlebars::{Handlebars, Helper, Context, RenderError, RenderContext, Output, HelperResult};
+use handlebars::{Handlebars, Helper, Context, RenderError, RenderContext, Output, HelperResult, TemplateError};
 
 use proxmox::tools::email::sendmail;
 use proxmox::api::schema::parse_property_string;
+use proxmox::try_block;
 
 use crate::{
     config::datastore::DataStoreConfig,
@@ -181,25 +182,33 @@ lazy_static::lazy_static!{
 
     static ref HANDLEBARS: Handlebars<'static> = {
         let mut hb = Handlebars::new();
+        let result: Result<(), TemplateError> = try_block!({
 
-        hb.set_strict_mode(true);
+            hb.set_strict_mode(true);
 
-        hb.register_helper("human-bytes", Box::new(handlebars_humam_bytes_helper));
-        hb.register_helper("relative-percentage", Box::new(handlebars_relative_percentage_helper));
+            hb.register_helper("human-bytes", Box::new(handlebars_humam_bytes_helper));
+            hb.register_helper("relative-percentage", Box::new(handlebars_relative_percentage_helper));
 
-        hb.register_template_string("gc_ok_template", GC_OK_TEMPLATE).unwrap();
-        hb.register_template_string("gc_err_template", GC_ERR_TEMPLATE).unwrap();
+            hb.register_template_string("gc_ok_template", GC_OK_TEMPLATE)?;
+            hb.register_template_string("gc_err_template", GC_ERR_TEMPLATE)?;
 
-        hb.register_template_string("verify_ok_template", VERIFY_OK_TEMPLATE).unwrap();
-        hb.register_template_string("verify_err_template", VERIFY_ERR_TEMPLATE).unwrap();
+            hb.register_template_string("verify_ok_template", VERIFY_OK_TEMPLATE)?;
+            hb.register_template_string("verify_err_template", VERIFY_ERR_TEMPLATE)?;
 
-        hb.register_template_string("sync_ok_template", SYNC_OK_TEMPLATE).unwrap();
-        hb.register_template_string("sync_err_template", SYNC_ERR_TEMPLATE).unwrap();
+            hb.register_template_string("sync_ok_template", SYNC_OK_TEMPLATE)?;
+            hb.register_template_string("sync_err_template", SYNC_ERR_TEMPLATE)?;
 
-        hb.register_template_string("tape_backup_ok_template", TAPE_BACKUP_OK_TEMPLATE).unwrap();
-        hb.register_template_string("tape_backup_err_template", TAPE_BACKUP_ERR_TEMPLATE).unwrap();
+            hb.register_template_string("tape_backup_ok_template", TAPE_BACKUP_OK_TEMPLATE)?;
+            hb.register_template_string("tape_backup_err_template", TAPE_BACKUP_ERR_TEMPLATE)?;
 
-        hb.register_template_string("package_update_template", PACKAGE_UPDATES_TEMPLATE).unwrap();
+            hb.register_template_string("package_update_template", PACKAGE_UPDATES_TEMPLATE)?;
+
+            Ok(())
+        });
+
+        if let Err(err) = result {
+            eprintln!("error during template registration: {}", err);
+        }
 
         hb
     };
@@ -600,3 +609,23 @@ fn handlebars_relative_percentage_helper(
     }
     Ok(())
 }
+
+#[test]
+fn test_template_register() {
+    HANDLEBARS.get_helper("human-bytes").unwrap();
+    HANDLEBARS.get_helper("relative-percentage").unwrap();
+
+    assert!(HANDLEBARS.has_template("gc_ok_template"));
+    assert!(HANDLEBARS.has_template("gc_err_template"));
+
+    assert!(HANDLEBARS.has_template("verify_ok_template"));
+    assert!(HANDLEBARS.has_template("verify_err_template"));
+
+    assert!(HANDLEBARS.has_template("sync_ok_template"));
+    assert!(HANDLEBARS.has_template("sync_err_template"));
+
+    assert!(HANDLEBARS.has_template("tape_backup_ok_template"));
+    assert!(HANDLEBARS.has_template("tape_backup_err_template"));
+
+    assert!(HANDLEBARS.has_template("package_update_template"));
+}
-- 
2.20.1





^ permalink raw reply	[flat|nested] 8+ messages in thread

* [pbs-devel] [PATCH proxmox-backup v2 3/5] server/email_notifications: do not double html escape
  2021-03-18 12:01 [pbs-devel] [PATCH proxmox-backup v2 0/5] improve tape backup and notifications Dominik Csapak
  2021-03-18 12:01 ` [pbs-devel] [PATCH proxmox-backup v2 1/5] tools/systemd/time: implement some Traits for TimeSpan Dominik Csapak
  2021-03-18 12:01 ` [pbs-devel] [PATCH proxmox-backup v2 2/5] server/email_notifications: do not panic on template registration Dominik Csapak
@ 2021-03-18 12:01 ` Dominik Csapak
  2021-03-18 12:01 ` [pbs-devel] [PATCH proxmox-backup v2 4/5] api2/tape/backup: include a summary on notification e-mails Dominik Csapak
                   ` (2 subsequent siblings)
  5 siblings, 0 replies; 8+ messages in thread
From: Dominik Csapak @ 2021-03-18 12:01 UTC (permalink / raw)
  To: pbs-devel

the default escape handler is handlebars::html_escape, but this are
plain text emails and we manually escape them for the html part, so
set the default escape handler to 'no_escape'

this avoids double html escape for the characters: '&"<>' in emails

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 src/server/email_notifications.rs | 1 +
 1 file changed, 1 insertion(+)

diff --git a/src/server/email_notifications.rs b/src/server/email_notifications.rs
index cd5a43fe..e92c8091 100644
--- a/src/server/email_notifications.rs
+++ b/src/server/email_notifications.rs
@@ -185,6 +185,7 @@ lazy_static::lazy_static!{
         let result: Result<(), TemplateError> = try_block!({
 
             hb.set_strict_mode(true);
+            hb.register_escape_fn(handlebars::no_escape);
 
             hb.register_helper("human-bytes", Box::new(handlebars_humam_bytes_helper));
             hb.register_helper("relative-percentage", Box::new(handlebars_relative_percentage_helper));
-- 
2.20.1





^ permalink raw reply	[flat|nested] 8+ messages in thread

* [pbs-devel] [PATCH proxmox-backup v2 4/5] api2/tape/backup: include a summary on notification e-mails
  2021-03-18 12:01 [pbs-devel] [PATCH proxmox-backup v2 0/5] improve tape backup and notifications Dominik Csapak
                   ` (2 preceding siblings ...)
  2021-03-18 12:01 ` [pbs-devel] [PATCH proxmox-backup v2 3/5] server/email_notifications: do not double html escape Dominik Csapak
@ 2021-03-18 12:01 ` Dominik Csapak
  2021-03-19  6:07   ` Dietmar Maurer
  2021-03-18 12:01 ` [pbs-devel] [PATCH proxmox-backup v2 5/5] api2/tape/backup: wait indefinitely for lock in scheduled backup jobs Dominik Csapak
  2021-03-19  6:09 ` [pbs-devel] [PATCH proxmox-backup v2 0/5] improve tape backup and notifications Dietmar Maurer
  5 siblings, 1 reply; 8+ messages in thread
From: Dominik Csapak @ 2021-03-18 12:01 UTC (permalink / raw)
  To: pbs-devel

for now only contains the list of included snapshots (if any),
as well as the backup duration

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 src/api2/tape/backup.rs           | 32 +++++++++++++++++++++++++------
 src/api2/types/tape/mod.rs        |  9 +++++++++
 src/server/email_notifications.rs | 13 +++++++++++++
 3 files changed, 48 insertions(+), 6 deletions(-)

diff --git a/src/api2/tape/backup.rs b/src/api2/tape/backup.rs
index 690b6855..0d42c31a 100644
--- a/src/api2/tape/backup.rs
+++ b/src/api2/tape/backup.rs
@@ -51,6 +51,7 @@ use crate::{
         JOB_ID_SCHEMA,
         MediaPoolConfig,
         Userid,
+        TapeBackupJobSummary,
     },
     server::WorkerTask,
     task::TaskState,
@@ -198,13 +199,16 @@ pub fn do_tape_backup_job(
             let notify_user = setup.notify_user.as_ref().unwrap_or_else(|| &Userid::root_userid());
             let email = lookup_user_email(notify_user);
 
-            let job_result = backup_worker(
+            let (job_result, summary) = match backup_worker(
                 &worker,
                 datastore,
                 &pool_config,
                 &setup,
                 email.clone(),
-            );
+            ) {
+                Ok(summary) => (Ok(()), summary),
+                Err(err) => (Err(err), Default::default()),
+            };
 
             let status = worker.create_state(&job_result);
 
@@ -214,6 +218,7 @@ pub fn do_tape_backup_job(
                     Some(job.jobname()),
                     &setup,
                     &job_result,
+                    summary,
                 ) {
                     eprintln!("send tape backup notification failed: {}", err);
                 }
@@ -340,13 +345,17 @@ pub fn backup(
         move |worker| {
             let _drive_lock = drive_lock; // keep lock guard
             set_tape_device_state(&setup.drive, &worker.upid().to_string())?;
-            let job_result = backup_worker(
+
+            let (job_result, summary) = match backup_worker(
                 &worker,
                 datastore,
                 &pool_config,
                 &setup,
                 email.clone(),
-            );
+            ) {
+                Ok(summary) => (Ok(()), summary),
+                Err(err) => (Err(err), Default::default()),
+            };
 
             if let Some(email) = email {
                 if let Err(err) = crate::server::send_tape_backup_status(
@@ -354,6 +363,7 @@ pub fn backup(
                     None,
                     &setup,
                     &job_result,
+                    summary,
                 ) {
                     eprintln!("send tape backup notification failed: {}", err);
                 }
@@ -374,9 +384,11 @@ fn backup_worker(
     pool_config: &MediaPoolConfig,
     setup: &TapeBackupJobSetup,
     email: Option<String>,
-) -> Result<(), Error> {
+) -> Result<TapeBackupJobSummary, Error> {
 
     let status_path = Path::new(TAPE_STATUS_DIR);
+    let start = std::time::Instant::now();
+    let mut summary: TapeBackupJobSummary = Default::default();
 
     let _lock = MediaPool::lock(status_path, &pool_config.name)?;
 
@@ -422,8 +434,11 @@ fn backup_worker(
                     task_log!(worker, "skip snapshot {}", info.backup_dir);
                     continue;
                 }
+                let snapshot_name = info.backup_dir.to_string();
                 if !backup_snapshot(worker, &mut pool_writer, datastore.clone(), info.backup_dir)? {
                     errors = true;
+                } else {
+                    summary.snapshot_list.push(snapshot_name);
                 }
                 progress.done_snapshots = 1;
                 task_log!(
@@ -439,8 +454,11 @@ fn backup_worker(
                     task_log!(worker, "skip snapshot {}", info.backup_dir);
                     continue;
                 }
+                let snapshot_name = info.backup_dir.to_string();
                 if !backup_snapshot(worker, &mut pool_writer, datastore.clone(), info.backup_dir)? {
                     errors = true;
+                } else {
+                    summary.snapshot_list.push(snapshot_name);
                 }
                 progress.done_snapshots = snapshot_number as u64 + 1;
                 task_log!(
@@ -464,7 +482,9 @@ fn backup_worker(
         bail!("Tape backup finished with some errors. Please check the task log.");
     }
 
-    Ok(())
+    summary.duration = start.elapsed();
+
+    Ok(summary)
 }
 
 // Try to update the the media online status
diff --git a/src/api2/types/tape/mod.rs b/src/api2/types/tape/mod.rs
index 68b2cf12..44c2236b 100644
--- a/src/api2/types/tape/mod.rs
+++ b/src/api2/types/tape/mod.rs
@@ -20,3 +20,12 @@ pub use media_location::*;
 
 mod media;
 pub use media::*;
+
+/// Summary of a successful Tape Job
+#[derive(Default)]
+pub struct TapeBackupJobSummary {
+    /// The list of snaphots backed up
+    pub snapshot_list: Vec<String>,
+    /// The total time of the backup job
+    pub duration: std::time::Duration,
+}
diff --git a/src/server/email_notifications.rs b/src/server/email_notifications.rs
index e92c8091..26002d39 100644
--- a/src/server/email_notifications.rs
+++ b/src/server/email_notifications.rs
@@ -18,6 +18,7 @@ use crate::{
         Userid,
         Notify,
         DatastoreNotify,
+        TapeBackupJobSummary,
     },
     tools::format::HumanByte,
 };
@@ -149,6 +150,14 @@ Datastore:  {{job.store}}
 Tape Pool:  {{job.pool}}
 Tape Drive: {{job.drive}}
 
+{{#if snapshot-list ~}}
+Snapshots included:
+
+{{#each snapshot-list~}}
+{{this}}
+{{/each~}}
+{{/if}}
+Duration: {{duration}}
 
 Tape Backup successful.
 
@@ -412,14 +421,18 @@ pub fn send_tape_backup_status(
     id: Option<&str>,
     job: &TapeBackupJobSetup,
     result: &Result<(), Error>,
+    summary: TapeBackupJobSummary,
 ) -> Result<(), Error> {
 
     let (fqdn, port) = get_server_url();
+    let duration: crate::tools::systemd::time::TimeSpan = summary.duration.into();
     let mut data = json!({
         "job": job,
         "fqdn": fqdn,
         "port": port,
         "id": id,
+        "snapshot-list": summary.snapshot_list,
+        "duration": duration.to_string(),
     });
 
     let text = match result {
-- 
2.20.1





^ permalink raw reply	[flat|nested] 8+ messages in thread

* [pbs-devel] [PATCH proxmox-backup v2 5/5] api2/tape/backup: wait indefinitely for lock in scheduled backup jobs
  2021-03-18 12:01 [pbs-devel] [PATCH proxmox-backup v2 0/5] improve tape backup and notifications Dominik Csapak
                   ` (3 preceding siblings ...)
  2021-03-18 12:01 ` [pbs-devel] [PATCH proxmox-backup v2 4/5] api2/tape/backup: include a summary on notification e-mails Dominik Csapak
@ 2021-03-18 12:01 ` Dominik Csapak
  2021-03-19  6:09 ` [pbs-devel] [PATCH proxmox-backup v2 0/5] improve tape backup and notifications Dietmar Maurer
  5 siblings, 0 replies; 8+ messages in thread
From: Dominik Csapak @ 2021-03-18 12:01 UTC (permalink / raw)
  To: pbs-devel

so that a user can schedule multiple backup jobs onto a single
media pool without having to consider timing them apart

this makes sense since we can backup multiple datastores onto
the same media-set but can only specify one datastore per backup job

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 src/api2/tape/backup.rs | 57 +++++++++++++++++++++++++++--------------
 1 file changed, 38 insertions(+), 19 deletions(-)

diff --git a/src/api2/tape/backup.rs b/src/api2/tape/backup.rs
index 0d42c31a..f29eb715 100644
--- a/src/api2/tape/backup.rs
+++ b/src/api2/tape/backup.rs
@@ -5,6 +5,7 @@ use anyhow::{bail, format_err, Error};
 use serde_json::Value;
 
 use proxmox::{
+    try_block,
     api::{
         api,
         RpcEnvironment,
@@ -177,8 +178,15 @@ pub fn do_tape_backup_job(
 
     let (drive_config, _digest) = config::drive::config()?;
 
-    // early check/lock before starting worker
-    let drive_lock = lock_tape_device(&drive_config, &setup.drive)?;
+    // for scheduled jobs we acquire the lock later in the worker
+    let drive_lock = if schedule.is_some() {
+        None
+    } else {
+        Some(lock_tape_device(&drive_config, &setup.drive)?)
+    };
+
+    let notify_user = setup.notify_user.as_ref().unwrap_or_else(|| &Userid::root_userid());
+    let email = lookup_user_email(notify_user);
 
     let upid_str = WorkerTask::new_thread(
         &worker_type,
@@ -186,26 +194,37 @@ pub fn do_tape_backup_job(
         auth_id.clone(),
         false,
         move |worker| {
-            let _drive_lock = drive_lock; // keep lock guard
-
-            set_tape_device_state(&setup.drive, &worker.upid().to_string())?;
             job.start(&worker.upid().to_string())?;
+            let mut drive_lock = drive_lock;
+
+            let (job_result, summary) = match try_block!({
+                if schedule.is_some() {
+                    // for scheduled tape backup jobs, we wait indefinitely for the lock
+                    task_log!(worker, "waiting for drive lock...");
+                    loop {
+                        if let Ok(lock) = lock_tape_device(&drive_config, &setup.drive) {
+                            drive_lock = Some(lock);
+                            break;
+                        } // ignore errors
+
+                        worker.check_abort()?;
+                    }
+                }
+                set_tape_device_state(&setup.drive, &worker.upid().to_string())?;
 
-            task_log!(worker,"Starting tape backup job '{}'", job_id);
-            if let Some(event_str) = schedule {
-                task_log!(worker,"task triggered by schedule '{}'", event_str);
-            }
-
-            let notify_user = setup.notify_user.as_ref().unwrap_or_else(|| &Userid::root_userid());
-            let email = lookup_user_email(notify_user);
+                task_log!(worker,"Starting tape backup job '{}'", job_id);
+                if let Some(event_str) = schedule {
+                    task_log!(worker,"task triggered by schedule '{}'", event_str);
+                }
 
-            let (job_result, summary) = match backup_worker(
-                &worker,
-                datastore,
-                &pool_config,
-                &setup,
-                email.clone(),
-            ) {
+                backup_worker(
+                    &worker,
+                    datastore,
+                    &pool_config,
+                    &setup,
+                    email.clone(),
+                )
+            }) {
                 Ok(summary) => (Ok(()), summary),
                 Err(err) => (Err(err), Default::default()),
             };
-- 
2.20.1





^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [pbs-devel] [PATCH proxmox-backup v2 4/5] api2/tape/backup: include a summary on notification e-mails
  2021-03-18 12:01 ` [pbs-devel] [PATCH proxmox-backup v2 4/5] api2/tape/backup: include a summary on notification e-mails Dominik Csapak
@ 2021-03-19  6:07   ` Dietmar Maurer
  0 siblings, 0 replies; 8+ messages in thread
From: Dietmar Maurer @ 2021-03-19  6:07 UTC (permalink / raw)
  To: Proxmox Backup Server development discussion, Dominik Csapak

comments inline

> On 03/18/2021 1:01 PM Dominik Csapak <d.csapak@proxmox.com> wrote:
> 
>  
> for now only contains the list of included snapshots (if any),
> as well as the backup duration
> 
> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
> ---
>  src/api2/tape/backup.rs           | 32 +++++++++++++++++++++++++------
>  src/api2/types/tape/mod.rs        |  9 +++++++++
>  src/server/email_notifications.rs | 13 +++++++++++++
>  3 files changed, 48 insertions(+), 6 deletions(-)
> 
> diff --git a/src/api2/tape/backup.rs b/src/api2/tape/backup.rs
> index 690b6855..0d42c31a 100644
> --- a/src/api2/tape/backup.rs
> +++ b/src/api2/tape/backup.rs
> @@ -51,6 +51,7 @@ use crate::{
>          JOB_ID_SCHEMA,
>          MediaPoolConfig,
>          Userid,
> +        TapeBackupJobSummary,
>      },
>      server::WorkerTask,
>      task::TaskState,
> @@ -198,13 +199,16 @@ pub fn do_tape_backup_job(
>              let notify_user = setup.notify_user.as_ref().unwrap_or_else(|| &Userid::root_userid());
>              let email = lookup_user_email(notify_user);
>  
> -            let job_result = backup_worker(
> +            let (job_result, summary) = match backup_worker(
>                  &worker,
>                  datastore,
>                  &pool_config,
>                  &setup,
>                  email.clone(),
> -            );
> +            ) {
> +                Ok(summary) => (Ok(()), summary),
> +                Err(err) => (Err(err), Default::default()),
> +            };
>  
>              let status = worker.create_state(&job_result);
>  
> @@ -214,6 +218,7 @@ pub fn do_tape_backup_job(
>                      Some(job.jobname()),
>                      &setup,
>                      &job_result,
> +                    summary,
>                  ) {
>                      eprintln!("send tape backup notification failed: {}", err);
>                  }
> @@ -340,13 +345,17 @@ pub fn backup(
>          move |worker| {
>              let _drive_lock = drive_lock; // keep lock guard
>              set_tape_device_state(&setup.drive, &worker.upid().to_string())?;
> -            let job_result = backup_worker(
> +
> +            let (job_result, summary) = match backup_worker(
>                  &worker,
>                  datastore,
>                  &pool_config,
>                  &setup,
>                  email.clone(),
> -            );
> +            ) {
> +                Ok(summary) => (Ok(()), summary),
> +                Err(err) => (Err(err), Default::default()),
> +            };
>  
>              if let Some(email) = email {
>                  if let Err(err) = crate::server::send_tape_backup_status(
> @@ -354,6 +363,7 @@ pub fn backup(
>                      None,
>                      &setup,
>                      &job_result,
> +                    summary,
>                  ) {
>                      eprintln!("send tape backup notification failed: {}", err);
>                  }
> @@ -374,9 +384,11 @@ fn backup_worker(
>      pool_config: &MediaPoolConfig,
>      setup: &TapeBackupJobSetup,
>      email: Option<String>,
> -) -> Result<(), Error> {
> +) -> Result<TapeBackupJobSummary, Error> {
>  
>      let status_path = Path::new(TAPE_STATUS_DIR);
> +    let start = std::time::Instant::now();
> +    let mut summary: TapeBackupJobSummary = Default::default();
>  
>      let _lock = MediaPool::lock(status_path, &pool_config.name)?;
>  
> @@ -422,8 +434,11 @@ fn backup_worker(
>                      task_log!(worker, "skip snapshot {}", info.backup_dir);
>                      continue;
>                  }
> +                let snapshot_name = info.backup_dir.to_string();
>                  if !backup_snapshot(worker, &mut pool_writer, datastore.clone(), info.backup_dir)? {
>                      errors = true;
> +                } else {
> +                    summary.snapshot_list.push(snapshot_name);
>                  }
>                  progress.done_snapshots = 1;
>                  task_log!(
> @@ -439,8 +454,11 @@ fn backup_worker(
>                      task_log!(worker, "skip snapshot {}", info.backup_dir);
>                      continue;
>                  }
> +                let snapshot_name = info.backup_dir.to_string();
>                  if !backup_snapshot(worker, &mut pool_writer, datastore.clone(), info.backup_dir)? {
>                      errors = true;
> +                } else {
> +                    summary.snapshot_list.push(snapshot_name);
>                  }
>                  progress.done_snapshots = snapshot_number as u64 + 1;
>                  task_log!(
> @@ -464,7 +482,9 @@ fn backup_worker(
>          bail!("Tape backup finished with some errors. Please check the task log.");
>      }
>  
> -    Ok(())
> +    summary.duration = start.elapsed();
> +
> +    Ok(summary)
>  }
>  
>  // Try to update the the media online status
> diff --git a/src/api2/types/tape/mod.rs b/src/api2/types/tape/mod.rs
> index 68b2cf12..44c2236b 100644
> --- a/src/api2/types/tape/mod.rs
> +++ b/src/api2/types/tape/mod.rs
> @@ -20,3 +20,12 @@ pub use media_location::*;
>  
>  mod media;
>  pub use media::*;
> +
> +/// Summary of a successful Tape Job
> +#[derive(Default)]
> +pub struct TapeBackupJobSummary {
> +    /// The list of snaphots backed up
> +    pub snapshot_list: Vec<String>,
> +    /// The total time of the backup job
> +    pub duration: std::time::Duration,
> +}

This is not an API type, and is not use in the API at all!
Please can  you define this type somewhere else?




^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [pbs-devel] [PATCH proxmox-backup v2 0/5] improve tape backup and notifications
  2021-03-18 12:01 [pbs-devel] [PATCH proxmox-backup v2 0/5] improve tape backup and notifications Dominik Csapak
                   ` (4 preceding siblings ...)
  2021-03-18 12:01 ` [pbs-devel] [PATCH proxmox-backup v2 5/5] api2/tape/backup: wait indefinitely for lock in scheduled backup jobs Dominik Csapak
@ 2021-03-19  6:09 ` Dietmar Maurer
  5 siblings, 0 replies; 8+ messages in thread
From: Dietmar Maurer @ 2021-03-19  6:09 UTC (permalink / raw)
  To: Proxmox Backup Server development discussion, Dominik Csapak

applied patch 1, 2 an d 3

please rework 4, and rebase 5 (does not apply anymore)




^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2021-03-19  6:09 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-18 12:01 [pbs-devel] [PATCH proxmox-backup v2 0/5] improve tape backup and notifications Dominik Csapak
2021-03-18 12:01 ` [pbs-devel] [PATCH proxmox-backup v2 1/5] tools/systemd/time: implement some Traits for TimeSpan Dominik Csapak
2021-03-18 12:01 ` [pbs-devel] [PATCH proxmox-backup v2 2/5] server/email_notifications: do not panic on template registration Dominik Csapak
2021-03-18 12:01 ` [pbs-devel] [PATCH proxmox-backup v2 3/5] server/email_notifications: do not double html escape Dominik Csapak
2021-03-18 12:01 ` [pbs-devel] [PATCH proxmox-backup v2 4/5] api2/tape/backup: include a summary on notification e-mails Dominik Csapak
2021-03-19  6:07   ` Dietmar Maurer
2021-03-18 12:01 ` [pbs-devel] [PATCH proxmox-backup v2 5/5] api2/tape/backup: wait indefinitely for lock in scheduled backup jobs Dominik Csapak
2021-03-19  6:09 ` [pbs-devel] [PATCH proxmox-backup v2 0/5] improve tape backup and notifications Dietmar Maurer

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal