* [pbs-devel] [PATCH v2 proxmox-backup 00/15] add job based verify scheduling
@ 2020-10-07 9:03 Hannes Laimer
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 01/15] add two new schemas for verify jobs Hannes Laimer
` (14 more replies)
0 siblings, 15 replies; 20+ messages in thread
From: Hannes Laimer @ 2020-10-07 9:03 UTC (permalink / raw)
To: pbs-devel
Replaces the first implementation of scheduled verification with a new
job-based version with additional options that may be specified through
the web ui.
Options available for verification jobs:
* schedule when to run the job
* set datastore on which the job should run
* set a number of days after which a verification becomes "outdated"
empty => verifications are valid forever
* specify if already successfuly verified snapshots should be verified
again even if they're not outdated(failed ones will always be done)
v2:
* added 'Ignire Verified' column in WebUI
* rebased onto master
* log number of planned verification
* ignore_verified is not optional anymore
* adjusted default column width for 'Days valid'
* failed verifications won't be verified again
* use proxmox::try_block! in worker
Hannes Laimer (15):
add two new schemas for verify jobs
add verify job config
api2: add verify job config endpoint
add do_verification_job function to verify.rs
api2: add verify job admin endpoint
add scheduling for verify jobs
set a diffrent worker_type based on what is going to be
verified(snapshot, group, ds)
ui: add verify job view
ui: add verify job edit window
ui: add task descriptions for the different types of verify(job,
snapshot, group, ds)
remove verify_schedule field from DatastoreConfig
remove verify_schedule field from datastore config endpoint
remove verify-schedule field from DataStoreEdit and DataStoreConfig
remove old verification scheduling from proxmox-backup-proxy.rs
verify job: log number of planned verifications
src/api2/admin.rs | 4 +-
src/api2/admin/datastore.rs | 5 +-
src/api2/admin/verify.rs | 107 ++++++++++++
src/api2/config.rs | 2 +
src/api2/config/datastore.rs | 24 ---
src/api2/config/verify.rs | 275 +++++++++++++++++++++++++++++++
src/api2/types/mod.rs | 10 ++
src/backup/verify.rs | 72 ++++++++
src/bin/proxmox-backup-proxy.rs | 115 ++++---------
src/config.rs | 1 +
src/config/datastore.rs | 6 -
src/config/verify.rs | 189 +++++++++++++++++++++
www/Makefile | 2 +
www/NavigationTree.js | 6 +
www/Utils.js | 5 +-
www/config/DataStoreConfig.js | 2 +-
www/config/VerifyView.js | 280 ++++++++++++++++++++++++++++++++
www/window/DataStoreEdit.js | 9 -
www/window/VerifyJobEdit.js | 89 ++++++++++
19 files changed, 1076 insertions(+), 127 deletions(-)
create mode 100644 src/api2/admin/verify.rs
create mode 100644 src/api2/config/verify.rs
create mode 100644 src/config/verify.rs
create mode 100644 www/config/VerifyView.js
create mode 100644 www/window/VerifyJobEdit.js
--
2.20.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [pbs-devel] [PATCH v2 proxmox-backup 01/15] add two new schemas for verify jobs
2020-10-07 9:03 [pbs-devel] [PATCH v2 proxmox-backup 00/15] add job based verify scheduling Hannes Laimer
@ 2020-10-07 9:03 ` Hannes Laimer
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 02/15] add verify job config Hannes Laimer
` (13 subsequent siblings)
14 siblings, 0 replies; 20+ messages in thread
From: Hannes Laimer @ 2020-10-07 9:03 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
src/api2/types/mod.rs | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/src/api2/types/mod.rs b/src/api2/types/mod.rs
index 348d3bf4..eabeab92 100644
--- a/src/api2/types/mod.rs
+++ b/src/api2/types/mod.rs
@@ -324,6 +324,16 @@ pub const REMOVE_VANISHED_BACKUPS_SCHEMA: Schema = BooleanSchema::new(
.default(true)
.schema();
+pub const IGNORE_VERIFIED_BACKUPS_SCHEMA: Schema = BooleanSchema::new(
+ "Do not verify backups that are already verified if their verification is not outdated.")
+ .default(true)
+ .schema();
+
+pub const VERIFICATION_OUTDATED_AFTER_SCHEMA: Schema =
+ IntegerSchema::new("Days after that a verification becomes outdated")
+ .minimum(1)
+ .schema();
+
pub const SINGLE_LINE_COMMENT_SCHEMA: Schema = StringSchema::new("Comment (single line).")
.format(&SINGLE_LINE_COMMENT_FORMAT)
.schema();
--
2.20.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [pbs-devel] [PATCH v2 proxmox-backup 02/15] add verify job config
2020-10-07 9:03 [pbs-devel] [PATCH v2 proxmox-backup 00/15] add job based verify scheduling Hannes Laimer
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 01/15] add two new schemas for verify jobs Hannes Laimer
@ 2020-10-07 9:03 ` Hannes Laimer
2020-10-08 7:50 ` Wolfgang Bumiller
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 03/15] api2: add verify job config endpoint Hannes Laimer
` (12 subsequent siblings)
14 siblings, 1 reply; 20+ messages in thread
From: Hannes Laimer @ 2020-10-07 9:03 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
src/config.rs | 1 +
src/config/verify.rs | 189 +++++++++++++++++++++++++++++++++++++++++++
2 files changed, 190 insertions(+)
create mode 100644 src/config/verify.rs
diff --git a/src/config.rs b/src/config.rs
index c2ac6da1..ab7fc81a 100644
--- a/src/config.rs
+++ b/src/config.rs
@@ -23,6 +23,7 @@ pub mod network;
pub mod remote;
pub mod sync;
pub mod user;
+pub mod verify;
/// Check configuration directory permissions
///
diff --git a/src/config/verify.rs b/src/config/verify.rs
new file mode 100644
index 00000000..7c92a16e
--- /dev/null
+++ b/src/config/verify.rs
@@ -0,0 +1,189 @@
+use anyhow::{Error};
+use lazy_static::lazy_static;
+use std::collections::HashMap;
+use serde::{Serialize, Deserialize};
+
+use proxmox::api::{
+ api,
+ schema::*,
+ section_config::{
+ SectionConfig,
+ SectionConfigData,
+ SectionConfigPlugin,
+ }
+};
+
+use proxmox::tools::{fs::replace_file, fs::CreateOptions};
+
+use crate::api2::types::*;
+
+lazy_static! {
+ static ref CONFIG: SectionConfig = init();
+}
+
+
+#[api(
+ properties: {
+ id: {
+ schema: JOB_ID_SCHEMA,
+ },
+ store: {
+ schema: DATASTORE_SCHEMA,
+ },
+ "ignore-verified": {
+ optional: true,
+ schema: IGNORE_VERIFIED_BACKUPS_SCHEMA,
+ },
+ "outdated-after": {
+ optional: true,
+ schema: VERIFICATION_OUTDATED_AFTER_SCHEMA,
+ },
+ comment: {
+ optional: true,
+ schema: SINGLE_LINE_COMMENT_SCHEMA,
+ },
+ schedule: {
+ optional: true,
+ schema: VERIFY_SCHEDULE_SCHEMA,
+ },
+ }
+)]
+#[serde(rename_all="kebab-case")]
+#[derive(Serialize,Deserialize)]
+/// Verify Job
+pub struct VerifyJobConfig {
+ pub id: String,
+ pub store: String,
+ #[serde(skip_serializing_if="Option::is_none")]
+ pub ignore_verified: Option<bool>,
+ #[serde(skip_serializing_if="Option::is_none")]
+ pub outdated_after: Option<i64>,
+ #[serde(skip_serializing_if="Option::is_none")]
+ pub comment: Option<String>,
+ #[serde(skip_serializing_if="Option::is_none")]
+ pub schedule: Option<String>,
+}
+
+// FIXME: generate duplicate schemas/structs from one listing?
+#[api(
+ properties: {
+ id: {
+ schema: JOB_ID_SCHEMA,
+ },
+ store: {
+ schema: DATASTORE_SCHEMA,
+ },
+ "ignore-verified": {
+ optional: true,
+ schema: IGNORE_VERIFIED_BACKUPS_SCHEMA,
+ },
+ "outdated-after": {
+ optional: true,
+ schema: VERIFICATION_OUTDATED_AFTER_SCHEMA,
+ },
+ comment: {
+ optional: true,
+ schema: SINGLE_LINE_COMMENT_SCHEMA,
+ },
+ schedule: {
+ optional: true,
+ schema: VERIFY_SCHEDULE_SCHEMA,
+ },
+ "next-run": {
+ description: "Estimated time of the next run (UNIX epoch).",
+ optional: true,
+ type: Integer,
+ },
+ "last-run-state": {
+ description: "Result of the last run.",
+ optional: true,
+ type: String,
+ },
+ "last-run-upid": {
+ description: "Task UPID of the last run.",
+ optional: true,
+ type: String,
+ },
+ "last-run-endtime": {
+ description: "Endtime of the last run.",
+ optional: true,
+ type: Integer,
+ },
+ }
+)]
+#[serde(rename_all="kebab-case")]
+#[derive(Serialize,Deserialize)]
+/// Status of Verify Job
+pub struct VerifyJobStatus {
+ pub id: String,
+ pub store: String,
+ #[serde(skip_serializing_if="Option::is_none")]
+ pub ignore_verified: Option<bool>,
+ #[serde(skip_serializing_if="Option::is_none")]
+ pub outdated_after: Option<i64>,
+ #[serde(skip_serializing_if="Option::is_none")]
+ pub comment: Option<String>,
+ #[serde(skip_serializing_if="Option::is_none")]
+ pub schedule: Option<String>,
+ #[serde(skip_serializing_if="Option::is_none")]
+ pub next_run: Option<i64>,
+ #[serde(skip_serializing_if="Option::is_none")]
+ pub last_run_state: Option<String>,
+ #[serde(skip_serializing_if="Option::is_none")]
+ pub last_run_upid: Option<String>,
+ #[serde(skip_serializing_if="Option::is_none")]
+ pub last_run_endtime: Option<i64>,
+}
+
+
+fn init() -> SectionConfig {
+ let obj_schema = match VerifyJobConfig::API_SCHEMA {
+ Schema::Object(ref obj_schema) => obj_schema,
+ _ => unreachable!(),
+ };
+
+ let plugin = SectionConfigPlugin::new("verify".to_string(), Some(String::from("id")), obj_schema);
+ let mut config = SectionConfig::new(&JOB_ID_SCHEMA);
+ config.register_plugin(plugin);
+
+ config
+}
+
+pub const VERIFY_CFG_FILENAME: &str = "/etc/proxmox-backup/verify.cfg";
+pub const VERIFY_CFG_LOCKFILE: &str = "/etc/proxmox-backup/.verify.lck";
+
+pub fn config() -> Result<(SectionConfigData, [u8;32]), Error> {
+
+ let content = proxmox::tools::fs::file_read_optional_string(VERIFY_CFG_FILENAME)?;
+ let content = content.unwrap_or(String::from(""));
+
+ let digest = openssl::sha::sha256(content.as_bytes());
+ let data = CONFIG.parse(VERIFY_CFG_FILENAME, &content)?;
+ Ok((data, digest))
+}
+
+pub fn save_config(config: &SectionConfigData) -> Result<(), Error> {
+ let raw = CONFIG.write(VERIFY_CFG_FILENAME, &config)?;
+
+ let backup_user = crate::backup::backup_user()?;
+ let mode = nix::sys::stat::Mode::from_bits_truncate(0o0640);
+ // set the correct owner/group/permissions while saving file
+ // owner(rw) = root, group(r)= backup
+
+ let options = CreateOptions::new()
+ .perm(mode)
+ .owner(nix::unistd::ROOT)
+ .group(backup_user.gid);
+
+ replace_file(VERIFY_CFG_FILENAME, raw.as_bytes(), options)?;
+
+ Ok(())
+}
+
+// shell completion helper
+pub fn complete_verify_job_id(_arg: &str, _param: &HashMap<String, String>) -> Vec<String> {
+ match config() {
+ Ok((data, _digest)) => data.sections.iter().map(|(id, _)| id.to_string()).collect(),
+ Err(_) => return vec![],
+ }
+}
\ No newline at end of file
--
2.20.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [pbs-devel] [PATCH v2 proxmox-backup 03/15] api2: add verify job config endpoint
2020-10-07 9:03 [pbs-devel] [PATCH v2 proxmox-backup 00/15] add job based verify scheduling Hannes Laimer
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 01/15] add two new schemas for verify jobs Hannes Laimer
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 02/15] add verify job config Hannes Laimer
@ 2020-10-07 9:03 ` Hannes Laimer
2020-10-08 8:22 ` Wolfgang Bumiller
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 04/15] add do_verification_job function to verify.rs Hannes Laimer
` (11 subsequent siblings)
14 siblings, 1 reply; 20+ messages in thread
From: Hannes Laimer @ 2020-10-07 9:03 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
src/api2/config.rs | 2 +
src/api2/config/verify.rs | 275 ++++++++++++++++++++++++++++++++++++++
2 files changed, 277 insertions(+)
create mode 100644 src/api2/config/verify.rs
diff --git a/src/api2/config.rs b/src/api2/config.rs
index be7397c8..7a5129c7 100644
--- a/src/api2/config.rs
+++ b/src/api2/config.rs
@@ -4,11 +4,13 @@ use proxmox::list_subdirs_api_method;
pub mod datastore;
pub mod remote;
pub mod sync;
+pub mod verify;
const SUBDIRS: SubdirMap = &[
("datastore", &datastore::ROUTER),
("remote", &remote::ROUTER),
("sync", &sync::ROUTER),
+ ("verify", &verify::ROUTER)
];
pub const ROUTER: Router = Router::new()
diff --git a/src/api2/config/verify.rs b/src/api2/config/verify.rs
new file mode 100644
index 00000000..9897aaf9
--- /dev/null
+++ b/src/api2/config/verify.rs
@@ -0,0 +1,275 @@
+use anyhow::{bail, Error};
+use serde_json::Value;
+use ::serde::{Deserialize, Serialize};
+
+use proxmox::api::{api, Router, RpcEnvironment};
+use proxmox::tools::fs::open_file_locked;
+
+use crate::api2::types::*;
+use crate::config::verify::{self, VerifyJobConfig};
+
+#[api(
+ input: {
+ properties: {},
+ },
+ returns: {
+ description: "List configured jobs.",
+ type: Array,
+ items: { type: verify::VerifyJobConfig },
+ },
+)]
+/// List all verify jobs
+pub fn list_verify_jobs(
+ _param: Value,
+ mut rpcenv: &mut dyn RpcEnvironment,
+) -> Result<Vec<VerifyJobConfig>, Error> {
+
+ let (config, digest) = verify::config()?;
+
+ let list = config.convert_to_typed_array("verify")?;
+
+ rpcenv["digest"] = proxmox::tools::digest_to_hex(&digest).into();
+
+ Ok(list)
+}
+
+
+#[api(
+ protected: true,
+ input: {
+ properties: {
+ id: {
+ schema: JOB_ID_SCHEMA,
+ },
+ store: {
+ schema: DATASTORE_SCHEMA,
+ },
+ "ignore-verified": {
+ optional: true,
+ schema: IGNORE_VERIFIED_BACKUPS_SCHEMA,
+ },
+ "outdated-after": {
+ optional: true,
+ schema: VERIFICATION_OUTDATED_AFTER_SCHEMA,
+ },
+ comment: {
+ optional: true,
+ schema: SINGLE_LINE_COMMENT_SCHEMA,
+ },
+ schedule: {
+ optional: true,
+ schema: VERIFY_SCHEDULE_SCHEMA,
+ },
+ }
+ }
+)]
+/// Create a new verify job.
+pub fn create_verify_job(param: Value) -> Result<(), Error> {
+
+ let _lock = open_file_locked(verify::VERIFY_CFG_LOCKFILE, std::time::Duration::new(10, 0), true)?;
+
+ let verify_job: verify::VerifyJobConfig = serde_json::from_value(param.clone())?;
+
+ let (mut config, _digest) = verify::config()?;
+
+ if let Some(_) = config.sections.get(&verify_job.id) {
+ bail!("job '{}' already exists.", verify_job.id);
+ }
+
+ config.set_data(&verify_job.id, "verify", &verify_job)?;
+
+ verify::save_config(&config)?;
+
+ crate::config::jobstate::create_state_file("verifyjob", &verify_job.id)?;
+
+ Ok(())
+}
+
+#[api(
+ input: {
+ properties: {
+ id: {
+ schema: JOB_ID_SCHEMA,
+ },
+ },
+ },
+ returns: {
+ description: "The verify job configuration.",
+ type: verify::VerifyJobConfig,
+ },
+)]
+/// Read a verify job configuration.
+pub fn read_verify_job(
+ id: String,
+ mut rpcenv: &mut dyn RpcEnvironment,
+) -> Result<VerifyJobConfig, Error> {
+ let (config, digest) = verify::config()?;
+
+ let verify_job = config.lookup("verify", &id)?;
+ rpcenv["digest"] = proxmox::tools::digest_to_hex(&digest).into();
+
+ Ok(verify_job)
+}
+
+#[api()]
+#[derive(Serialize, Deserialize)]
+#[serde(rename_all="kebab-case")]
+#[allow(non_camel_case_types)]
+/// Deletable property name
+pub enum DeletableProperty {
+ /// Delete ignore verified
+ ignore_verified,
+ /// Delete the comment property.
+ comment,
+ /// Delete the job schedule.
+ schedule,
+ /// Delete outdated_after.
+ outdated_after
+}
+
+#[api(
+ protected: true,
+ input: {
+ properties: {
+ id: {
+ schema: JOB_ID_SCHEMA,
+ },
+ store: {
+ optional: true,
+ schema: DATASTORE_SCHEMA,
+ },
+ "ignore-verified": {
+ optional: true,
+ schema: IGNORE_VERIFIED_BACKUPS_SCHEMA,
+ },
+ "outdated-after": {
+ optional: true,
+ schema: VERIFICATION_OUTDATED_AFTER_SCHEMA,
+ },
+ comment: {
+ optional: true,
+ schema: SINGLE_LINE_COMMENT_SCHEMA,
+ },
+ schedule: {
+ optional: true,
+ schema: VERIFY_SCHEDULE_SCHEMA,
+ },
+ delete: {
+ description: "List of properties to delete.",
+ type: Array,
+ optional: true,
+ items: {
+ type: DeletableProperty,
+ }
+ },
+ digest: {
+ optional: true,
+ schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
+ },
+ },
+ },
+)]
+/// Update verify job config.
+pub fn update_verify_job(
+ id: String,
+ store: Option<String>,
+ ignore_verified: Option<bool>,
+ outdated_after: Option<i64>,
+ comment: Option<String>,
+ schedule: Option<String>,
+ delete: Option<Vec<DeletableProperty>>,
+ digest: Option<String>,
+) -> Result<(), Error> {
+
+ let _lock = open_file_locked(verify::VERIFY_CFG_LOCKFILE, std::time::Duration::new(10, 0), true)?;
+
+ // pass/compare digest
+ let (mut config, expected_digest) = verify::config()?;
+
+ if let Some(ref digest) = digest {
+ let digest = proxmox::tools::hex_to_digest(digest)?;
+ crate::tools::detect_modified_configuration_file(&digest, &expected_digest)?;
+ }
+
+ let mut data: verify::VerifyJobConfig = config.lookup("verify", &id)?;
+
+ if let Some(delete) = delete {
+ for delete_prop in delete {
+ match delete_prop {
+ DeletableProperty::ignore_verified => { data.ignore_verified = None; },
+ DeletableProperty::outdated_after => { data.outdated_after = None; },
+ DeletableProperty::comment => { data.comment = None; },
+ DeletableProperty::schedule => { data.schedule = None; },
+ }
+ }
+ }
+
+ if let Some(comment) = comment {
+ let comment = comment.trim().to_string();
+ if comment.is_empty() {
+ data.comment = None;
+ } else {
+ data.comment = Some(comment);
+ }
+ }
+
+ if let Some(store) = store { data.store = store; }
+
+ if ignore_verified.is_some() { data.ignore_verified = ignore_verified; }
+ if outdated_after.is_some() { data.outdated_after = outdated_after; }
+ if schedule.is_some() { data.schedule = schedule; }
+
+ config.set_data(&id, "verify", &data)?;
+
+ verify::save_config(&config)?;
+
+ Ok(())
+}
+
+#[api(
+ protected: true,
+ input: {
+ properties: {
+ id: {
+ schema: JOB_ID_SCHEMA,
+ },
+ digest: {
+ optional: true,
+ schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
+ },
+ },
+ },
+)]
+/// Remove a verify job configuration
+pub fn delete_verify_job(id: String, digest: Option<String>) -> Result<(), Error> {
+
+ let _lock = open_file_locked(verify::VERIFY_CFG_LOCKFILE, std::time::Duration::new(10, 0), true)?;
+
+ let (mut config, expected_digest) = verify::config()?;
+
+ if let Some(ref digest) = digest {
+ let digest = proxmox::tools::hex_to_digest(digest)?;
+ crate::tools::detect_modified_configuration_file(&digest, &expected_digest)?;
+ }
+
+ match config.sections.get(&id) {
+ Some(_) => { config.sections.remove(&id); },
+ None => bail!("job '{}' does not exist.", id),
+ }
+
+ verify::save_config(&config)?;
+
+ crate::config::jobstate::remove_state_file("verifyjob", &id)?;
+
+ Ok(())
+}
+
+const ITEM_ROUTER: Router = Router::new()
+ .get(&API_METHOD_READ_VERIFY_JOB)
+ .put(&API_METHOD_UPDATE_VERIFY_JOB)
+ .delete(&API_METHOD_DELETE_VERIFY_JOB);
+
+pub const ROUTER: Router = Router::new()
+ .get(&API_METHOD_LIST_VERIFY_JOBS)
+ .post(&API_METHOD_CREATE_VERIFY_JOB)
+ .match_all("id", &ITEM_ROUTER);
\ No newline at end of file
--
2.20.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [pbs-devel] [PATCH v2 proxmox-backup 04/15] add do_verification_job function to verify.rs
2020-10-07 9:03 [pbs-devel] [PATCH v2 proxmox-backup 00/15] add job based verify scheduling Hannes Laimer
` (2 preceding siblings ...)
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 03/15] api2: add verify job config endpoint Hannes Laimer
@ 2020-10-07 9:03 ` Hannes Laimer
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 05/15] api2: add verify job admin endpoint Hannes Laimer
` (10 subsequent siblings)
14 siblings, 0 replies; 20+ messages in thread
From: Hannes Laimer @ 2020-10-07 9:03 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
src/backup/verify.rs | 71 ++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 71 insertions(+)
diff --git a/src/backup/verify.rs b/src/backup/verify.rs
index 0c55305f..517ecbb2 100644
--- a/src/backup/verify.rs
+++ b/src/backup/verify.rs
@@ -8,6 +8,8 @@ use anyhow::{bail, format_err, Error};
use crate::{
server::WorkerTask,
api2::types::*,
+ config::jobstate::Job,
+ config::verify::VerifyJobConfig,
tools::ParallelHandler,
backup::{
DataStore,
@@ -419,3 +421,72 @@ pub fn verify_all_backups(datastore: Arc<DataStore>, worker: Arc<WorkerTask>) ->
Ok(errors)
}
+
+/// Runs a verification job.
+pub fn do_verification_job(
+ mut job: Job,
+ verify_job: VerifyJobConfig,
+ userid: &Userid,
+ schedule: Option<String>,
+) -> Result<String, Error> {
+ let datastore = DataStore::lookup_datastore(&verify_job.store)?;
+ let mut backups_to_verify = BackupInfo::list_backups(&datastore.base_path())?;
+ if verify_job.ignore_verified.unwrap_or(true) {
+ backups_to_verify.retain(|backup_info| {
+ if let Ok((manifest, _)) = datastore.load_manifest(&backup_info.backup_dir) {
+ let verify = manifest.unprotected["verify_state"].clone();
+ if let Ok(verify) = serde_json::from_value::<SnapshotVerifyState>(verify) {
+ let days_since_last_verify =
+ (proxmox::tools::time::epoch_i64() - verify.upid.starttime) / 86400;
+ // if outdated_after is None, verifications do not become outdated
+ verify_job.outdated_after.is_some() && days_since_last_verify > verify_job.outdated_after.unwrap()
+ } else { true } // was never verified, therefore we always want to verify
+ } else { false } // manifest could not be loaded, do not verify in that case
+ })
+ }
+ let job_id = job.jobname().to_string();
+ let worker_type = job.jobtype().to_string();
+ let upid_str = WorkerTask::new_thread(
+ &worker_type,
+ Some(job.jobname().to_string()),
+ userid.clone(),
+ false,
+ move |worker| {
+ job.start(&worker.upid().to_string())?;
+ let verified_chunks = Arc::new(Mutex::new(HashSet::with_capacity(1024 * 16)));
+ let corrupt_chunks = Arc::new(Mutex::new(HashSet::with_capacity(64)));
+ worker.log(format!("Starting datastore verify job '{}'", job_id));
+ if let Some(event_str) = schedule {
+ worker.log(format!("task triggered by schedule '{}'", event_str));
+ }
+ let result = proxmox::try_block!({
+ let mut failed_dirs: Vec<String> = Vec::new();
+ for backup_info in backups_to_verify {
+ if let Ok(false) = verify_backup_dir(
+ datastore.clone(),
+ &backup_info.backup_dir,
+ verified_chunks.clone(),
+ corrupt_chunks.clone(),
+ worker.clone(),
+ ) { failed_dirs.push(backup_info.backup_dir.to_string()); }
+ }
+ if !failed_dirs.is_empty() {
+ worker.log("Failed to verify following snapshots:");
+ for dir in failed_dirs {
+ worker.log(format!("\t{}", dir));
+ }
+ bail!("verification failed - please check the log for details");
+ }
+ Ok(())
+ });
+
+ let status = worker.create_state(&result);
+
+ if let Err(err) = job.finish(status) {
+ eprintln!("could not finish job state for {}: {}", job.jobtype().to_string(), err);
+ }
+
+ result
+ })?;
+ Ok(upid_str)
+}
--
2.20.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [pbs-devel] [PATCH v2 proxmox-backup 05/15] api2: add verify job admin endpoint
2020-10-07 9:03 [pbs-devel] [PATCH v2 proxmox-backup 00/15] add job based verify scheduling Hannes Laimer
` (3 preceding siblings ...)
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 04/15] add do_verification_job function to verify.rs Hannes Laimer
@ 2020-10-07 9:03 ` Hannes Laimer
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 06/15] add scheduling for verify jobs Hannes Laimer
` (9 subsequent siblings)
14 siblings, 0 replies; 20+ messages in thread
From: Hannes Laimer @ 2020-10-07 9:03 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
src/api2/admin.rs | 4 +-
src/api2/admin/verify.rs | 107 +++++++++++++++++++++++++++++++++++++++
2 files changed, 110 insertions(+), 1 deletion(-)
create mode 100644 src/api2/admin/verify.rs
diff --git a/src/api2/admin.rs b/src/api2/admin.rs
index b927ce1e..79ce29f3 100644
--- a/src/api2/admin.rs
+++ b/src/api2/admin.rs
@@ -3,10 +3,12 @@ use proxmox::list_subdirs_api_method;
pub mod datastore;
pub mod sync;
+pub mod verify;
const SUBDIRS: SubdirMap = &[
("datastore", &datastore::ROUTER),
- ("sync", &sync::ROUTER)
+ ("sync", &sync::ROUTER),
+ ("verify", &verify::ROUTER)
];
pub const ROUTER: Router = Router::new()
diff --git a/src/api2/admin/verify.rs b/src/api2/admin/verify.rs
new file mode 100644
index 00000000..06262efb
--- /dev/null
+++ b/src/api2/admin/verify.rs
@@ -0,0 +1,107 @@
+use anyhow::{format_err, Error};
+
+use proxmox::api::router::SubdirMap;
+use proxmox::{list_subdirs_api_method, sortable};
+use proxmox::api::{api, ApiMethod, Router, RpcEnvironment};
+
+use crate::api2::types::*;
+use crate::backup::do_verification_job;
+use crate::config::jobstate::{Job, JobState};
+use crate::config::verify;
+use crate::config::verify::{VerifyJobConfig, VerifyJobStatus};
+use serde_json::Value;
+use crate::tools::systemd::time::{parse_calendar_event, compute_next_event};
+use crate::server::UPID;
+
+#[api(
+ input: {
+ properties: {},
+ },
+ returns: {
+ description: "List configured jobs and their status.",
+ type: Array,
+ items: { type: verify::VerifyJobStatus },
+ },
+)]
+/// List all verify jobs
+pub fn list_verify_jobs(
+ _param: Value,
+ mut rpcenv: &mut dyn RpcEnvironment,
+) -> Result<Vec<VerifyJobStatus>, Error> {
+
+ let (config, digest) = verify::config()?;
+
+ let mut list: Vec<VerifyJobStatus> = config.convert_to_typed_array("verify")?;
+
+ for job in &mut list {
+ let last_state = JobState::load("verifyjob", &job.id)
+ .map_err(|err| format_err!("could not open statefile for {}: {}", &job.id, err))?;
+
+ let (upid, endtime, state, starttime) = match last_state {
+ JobState::Created { time } => (None, None, None, time),
+ JobState::Started { upid } => {
+ let parsed_upid: UPID = upid.parse()?;
+ (Some(upid), None, None, parsed_upid.starttime)
+ },
+ JobState::Finished { upid, state } => {
+ let parsed_upid: UPID = upid.parse()?;
+ (Some(upid), Some(state.endtime()), Some(state.to_string()), parsed_upid.starttime)
+ },
+ };
+
+ job.last_run_upid = upid;
+ job.last_run_state = state;
+ job.last_run_endtime = endtime;
+
+ let last = job.last_run_endtime.unwrap_or_else(|| starttime);
+
+ job.next_run = (|| -> Option<i64> {
+ let schedule = job.schedule.as_ref()?;
+ let event = parse_calendar_event(&schedule).ok()?;
+ // ignore errors
+ compute_next_event(&event, last, false).unwrap_or_else(|_| None)
+ })();
+ }
+
+ rpcenv["digest"] = proxmox::tools::digest_to_hex(&digest).into();
+
+ Ok(list)
+}
+
+#[api(
+ input: {
+ properties: {
+ id: {
+ schema: JOB_ID_SCHEMA,
+ }
+ }
+ }
+)]
+/// Runs a verify job manually.
+fn run_verify_job(
+ id: String,
+ _info: &ApiMethod,
+ rpcenv: &mut dyn RpcEnvironment,
+) -> Result<String, Error> {
+ let (config, _digest) = verify::config()?;
+ let verify_job: VerifyJobConfig = config.lookup("verify", &id)?;
+
+ let userid: Userid = rpcenv.get_user().unwrap().parse()?;
+
+ let job = Job::new("verifyjob", &id)?;
+
+ let upid_str = do_verification_job(job, verify_job, &userid, None)?;
+
+ Ok(upid_str)
+}
+
+#[sortable]
+const VERIFY_INFO_SUBDIRS: SubdirMap = &[("run", &Router::new().post(&API_METHOD_RUN_VERIFY_JOB))];
+
+const VERIFY_INFO_ROUTER: Router = Router::new()
+ .get(&list_subdirs_api_method!(VERIFY_INFO_SUBDIRS))
+ .subdirs(VERIFY_INFO_SUBDIRS);
+
+pub const ROUTER: Router = Router::new()
+ .get(&API_METHOD_LIST_VERIFY_JOBS)
+ .match_all("id", &VERIFY_INFO_ROUTER);
--
2.20.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [pbs-devel] [PATCH v2 proxmox-backup 06/15] add scheduling for verify jobs
2020-10-07 9:03 [pbs-devel] [PATCH v2 proxmox-backup 00/15] add job based verify scheduling Hannes Laimer
` (4 preceding siblings ...)
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 05/15] api2: add verify job admin endpoint Hannes Laimer
@ 2020-10-07 9:03 ` Hannes Laimer
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 07/15] set a diffrent worker_type based on what is going to be verified(snapshot, group, ds) Hannes Laimer
` (8 subsequent siblings)
14 siblings, 0 replies; 20+ messages in thread
From: Hannes Laimer @ 2020-10-07 9:03 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
src/bin/proxmox-backup-proxy.rs | 62 +++++++++++++++++++++++++++++++++
1 file changed, 62 insertions(+)
diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
index 67fbc541..79d5fe50 100644
--- a/src/bin/proxmox-backup-proxy.rs
+++ b/src/bin/proxmox-backup-proxy.rs
@@ -19,6 +19,7 @@ use proxmox_backup::auth_helpers::*;
use proxmox_backup::tools::disks::{ DiskManage, zfs_pool_stats };
use proxmox_backup::api2::pull::do_sync_job;
+use proxmox_backup::backup::do_verification_job;
fn main() -> Result<(), Error> {
proxmox_backup::tools::setup_safe_path_env();
@@ -198,6 +199,7 @@ async fn schedule_tasks() -> Result<(), Error> {
schedule_datastore_prune().await;
schedule_datastore_verification().await;
schedule_datastore_sync_jobs().await;
+ schedule_datastore_verify_jobs().await;
schedule_task_log_rotate().await;
Ok(())
@@ -656,6 +658,66 @@ async fn schedule_datastore_sync_jobs() {
}
}
+async fn schedule_datastore_verify_jobs() {
+ use proxmox_backup::{
+ config::{ verify::{self, VerifyJobConfig}, jobstate::{self, Job} },
+ tools::systemd::time::{ parse_calendar_event, compute_next_event },
+ };
+ let config = match verify::config() {
+ Err(err) => {
+ eprintln!("unable to read verify job config - {}", err);
+ return;
+ }
+ Ok((config, _digest)) => config,
+ };
+ for (job_id, (_, job_config)) in config.sections {
+ let job_config: VerifyJobConfig = match serde_json::from_value(job_config) {
+ Ok(c) => c,
+ Err(err) => {
+ eprintln!("verify job config from_value failed - {}", err);
+ continue;
+ }
+ };
+ let event_str = match job_config.schedule {
+ Some(ref event_str) => event_str.clone(),
+ None => continue,
+ };
+ let event = match parse_calendar_event(&event_str) {
+ Ok(event) => event,
+ Err(err) => {
+ eprintln!("unable to parse schedule '{}' - {}", event_str, err);
+ continue;
+ }
+ };
+ let worker_type = "verifyjob";
+ let last = match jobstate::last_run_time(worker_type, &job_id) {
+ Ok(time) => time,
+ Err(err) => {
+ eprintln!("could not get last run time of {} {}: {}", worker_type, job_id, err);
+ continue;
+ }
+ };
+ let next = match compute_next_event(&event, last, false) {
+ Ok(Some(next)) => next,
+ Ok(None) => continue,
+ Err(err) => {
+ eprintln!("compute_next_event for '{}' failed - {}", event_str, err);
+ continue;
+ }
+ };
+ let now = proxmox::tools::time::epoch_i64();
+ if next > now { continue; }
+ let job = match Job::new(worker_type, &job_id) {
+ Ok(job) => job,
+ Err(_) => continue, // could not get lock
+ };
+ let userid = Userid::backup_userid().clone();
+ if let Err(err) = do_verification_job(job, job_config, &userid, Some(event_str)) {
+ eprintln!("unable to start datastore verify job {} - {}", &job_id, err);
+ }
+ }
+}
+
async fn schedule_task_log_rotate() {
use proxmox_backup::{
config::jobstate::{self, Job},
--
2.20.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [pbs-devel] [PATCH v2 proxmox-backup 07/15] set a diffrent worker_type based on what is going to be verified(snapshot, group, ds)
2020-10-07 9:03 [pbs-devel] [PATCH v2 proxmox-backup 00/15] add job based verify scheduling Hannes Laimer
` (5 preceding siblings ...)
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 06/15] add scheduling for verify jobs Hannes Laimer
@ 2020-10-07 9:03 ` Hannes Laimer
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 08/15] ui: add verify job view Hannes Laimer
` (7 subsequent siblings)
14 siblings, 0 replies; 20+ messages in thread
From: Hannes Laimer @ 2020-10-07 9:03 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
src/api2/admin/datastore.rs | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index af3af0ad..6ae606d9 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -486,17 +486,20 @@ pub fn verify(
let mut backup_dir = None;
let mut backup_group = None;
+ let mut worker_type = "verify";
match (backup_type, backup_id, backup_time) {
(Some(backup_type), Some(backup_id), Some(backup_time)) => {
worker_id = format!("{}_{}_{}_{:08X}", store, backup_type, backup_id, backup_time);
let dir = BackupDir::new(backup_type, backup_id, backup_time)?;
backup_dir = Some(dir);
+ worker_type = "verify_snapshot";
}
(Some(backup_type), Some(backup_id), None) => {
worker_id = format!("{}_{}_{}", store, backup_type, backup_id);
let group = BackupGroup::new(backup_type, backup_id);
backup_group = Some(group);
+ worker_type = "verify_group";
}
(None, None, None) => {
worker_id = store.clone();
@@ -508,7 +511,7 @@ pub fn verify(
let to_stdout = if rpcenv.env_type() == RpcEnvironmentType::CLI { true } else { false };
let upid_str = WorkerTask::new_thread(
- "verify",
+ worker_type,
Some(worker_id.clone()),
userid,
to_stdout,
--
2.20.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [pbs-devel] [PATCH v2 proxmox-backup 08/15] ui: add verify job view
2020-10-07 9:03 [pbs-devel] [PATCH v2 proxmox-backup 00/15] add job based verify scheduling Hannes Laimer
` (6 preceding siblings ...)
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 07/15] set a diffrent worker_type based on what is going to be verified(snapshot, group, ds) Hannes Laimer
@ 2020-10-07 9:03 ` Hannes Laimer
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 09/15] ui: add verify job edit window Hannes Laimer
` (6 subsequent siblings)
14 siblings, 0 replies; 20+ messages in thread
From: Hannes Laimer @ 2020-10-07 9:03 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
www/Makefile | 1 +
www/NavigationTree.js | 6 +
www/config/VerifyView.js | 280 +++++++++++++++++++++++++++++++++++++++
3 files changed, 287 insertions(+)
create mode 100644 www/config/VerifyView.js
diff --git a/www/Makefile b/www/Makefile
index 8e1ebbfc..a6c46718 100644
--- a/www/Makefile
+++ b/www/Makefile
@@ -16,6 +16,7 @@ JSSRC= \
config/RemoteView.js \
config/ACLView.js \
config/SyncView.js \
+ config/VerifyView.js \
config/DataStoreConfig.js \
window/UserEdit.js \
window/UserPassword.js \
diff --git a/www/NavigationTree.js b/www/NavigationTree.js
index 3fa782e1..a2835cd2 100644
--- a/www/NavigationTree.js
+++ b/www/NavigationTree.js
@@ -42,6 +42,12 @@ Ext.define('PBS.store.NavigationStore', {
path: 'pbsSyncJobView',
leaf: true,
},
+ {
+ text: gettext('Verify Jobs'),
+ iconCls: 'fa fa-check-circle',
+ path: 'pbsVerifyJobView',
+ leaf: true,
+ },
{
text: gettext('Subscription'),
iconCls: 'fa fa-support',
diff --git a/www/config/VerifyView.js b/www/config/VerifyView.js
new file mode 100644
index 00000000..91ceec46
--- /dev/null
+++ b/www/config/VerifyView.js
@@ -0,0 +1,280 @@
+Ext.define('pbs-verify-jobs-status', {
+ extend: 'Ext.data.Model',
+ fields: [
+ 'id', 'store', 'outdated-after', 'ignore-verified', 'schedule',
+ 'next-run', 'last-run-upid', 'last-run-state', 'last-run-endtime',
+ {
+ name: 'duration',
+ calculate: function(data) {
+ let endtime = data['last-run-endtime'];
+ if (!endtime) return undefined;
+ let task = Proxmox.Utils.parse_task_upid(data['last-run-upid']);
+ return endtime - task.starttime;
+ },
+ },
+ ],
+ idProperty: 'id',
+ proxy: {
+ type: 'proxmox',
+ url: '/api2/json/admin/verify',
+ },
+});
+
+Ext.define('PBS.config.VerifyJobView', {
+ extend: 'Ext.grid.GridPanel',
+ alias: 'widget.pbsVerifyJobView',
+
+ stateful: true,
+ stateId: 'grid-verify-jobs',
+
+ title: gettext('Verify Jobs'),
+
+ controller: {
+ xclass: 'Ext.app.ViewController',
+
+ addVerifyJob: function() {
+ let me = this;
+ Ext.create('PBS.window.VerifyJobEdit', {
+ listeners: {
+ destroy: function() {
+ me.reload();
+ },
+ },
+ }).show();
+ },
+
+ editVerifyJob: function() {
+ let me = this;
+ let view = me.getView();
+ let selection = view.getSelection();
+ if (selection.length < 1) return;
+
+ Ext.create('PBS.window.VerifyJobEdit', {
+ id: selection[0].data.id,
+ listeners: {
+ destroy: function() {
+ me.reload();
+ },
+ },
+ }).show();
+ },
+
+ openTaskLog: function() {
+ let me = this;
+ let view = me.getView();
+ let selection = view.getSelection();
+ if (selection.length < 1) return;
+
+ let upid = selection[0].data['last-run-upid'];
+ if (!upid) return;
+
+ Ext.create('Proxmox.window.TaskViewer', {
+ upid
+ }).show();
+ },
+
+ runVerifyJob: function() {
+ let me = this;
+ let view = me.getView();
+ let selection = view.getSelection();
+ if (selection.length < 1) return;
+
+ let id = selection[0].data.id;
+ Proxmox.Utils.API2Request({
+ method: 'POST',
+ url: `/admin/verify/${id}/run`,
+ success: function(response, opt) {
+ Ext.create('Proxmox.window.TaskViewer', {
+ upid: response.result.data,
+ taskDone: function(success) {
+ me.reload();
+ },
+ }).show();
+ },
+ failure: function(response, opt) {
+ Ext.Msg.alert(gettext('Error'), response.htmlStatus);
+ },
+ });
+ },
+
+ render_verify_status: function(value, metadata, record) {
+ if (!record.data['last-run-upid']) {
+ return '-';
+ }
+
+ if (!record.data['last-run-endtime']) {
+ metadata.tdCls = 'x-grid-row-loading';
+ return '';
+ }
+
+ let parsed = Proxmox.Utils.parse_task_status(value);
+ let text = value;
+ let icon = '';
+ switch (parsed) {
+ case 'unknown':
+ icon = 'question faded';
+ text = Proxmox.Utils.unknownText;
+ break;
+ case 'error':
+ icon = 'times critical';
+ text = Proxmox.Utils.errorText + ': ' + value;
+ break;
+ case 'warning':
+ icon = 'exclamation warning';
+ break;
+ case 'ok':
+ icon = 'check good';
+ text = gettext("OK");
+ }
+
+ return `<i class="fa fa-${icon}"></i> ${text}`;
+ },
+
+ render_next_run: function(value, metadat, record) {
+ if (!value) return '-';
+
+ let now = new Date();
+ let next = new Date(value*1000);
+
+ if (next < now) {
+ return gettext('pending');
+ }
+ return Proxmox.Utils.render_timestamp(value);
+ },
+
+ render_optional_timestamp: function(value, metadata, record) {
+ if (!value) return '-';
+ return Proxmox.Utils.render_timestamp(value);
+ },
+
+ reload: function() { this.getView().getStore().rstore.load(); },
+
+ init: function(view) {
+ Proxmox.Utils.monStoreErrors(view, view.getStore().rstore);
+ },
+ },
+
+ listeners: {
+ activate: 'reload',
+ itemdblclick: 'editVerifyJob',
+ },
+
+ store: {
+ type: 'diff',
+ autoDestroy: true,
+ autoDestroyRstore: true,
+ sorters: 'id',
+ rstore: {
+ type: 'update',
+ storeid: 'pbs-verify-jobs-status',
+ model: 'pbs-verify-jobs-status',
+ autoStart: true,
+ interval: 5000,
+ },
+ },
+
+ tbar: [
+ {
+ xtype: 'proxmoxButton',
+ text: gettext('Add'),
+ handler: 'addVerifyJob',
+ selModel: false,
+ },
+ {
+ xtype: 'proxmoxButton',
+ text: gettext('Edit'),
+ handler: 'editVerifyJob',
+ disabled: true,
+ },
+ {
+ xtype: 'proxmoxStdRemoveButton',
+ baseurl: '/config/verify/',
+ callback: 'reload',
+ },
+ '-',
+ {
+ xtype: 'proxmoxButton',
+ text: gettext('Log'),
+ handler: 'openTaskLog',
+ enableFn: (rec) => !!rec.data['last-run-upid'],
+ disabled: true,
+ },
+ {
+ xtype: 'proxmoxButton',
+ text: gettext('Run now'),
+ handler: 'runVerifyJob',
+ disabled: true,
+ },
+ ],
+
+ viewConfig: {
+ trackOver: false,
+ },
+
+ columns: [
+ {
+ header: gettext('Verify Job'),
+ width: 100,
+ sortable: true,
+ renderer: Ext.String.htmlEncode,
+ dataIndex: 'id',
+ },
+ {
+ header: gettext('Datastore'),
+ width: 100,
+ sortable: true,
+ dataIndex: 'store',
+ },
+ {
+ header: gettext('Days valid'),
+ width: 125,
+ sortable: true,
+ dataIndex: 'outdated-after',
+ },
+ {
+ header: gettext('Ignore verified'),
+ width: 125,
+ sortable: true,
+ renderer: Proxmox.Utils.format_boolean,
+ dataIndex: 'ignore-verified',
+ },
+ {
+ header: gettext('Schedule'),
+ sortable: true,
+ dataIndex: 'schedule',
+ },
+ {
+ header: gettext('Status'),
+ dataIndex: 'last-run-state',
+ flex: 1,
+ renderer: 'render_verify_status',
+ },
+ {
+ header: gettext('Last Verification'),
+ sortable: true,
+ minWidth: 200,
+ renderer: 'render_optional_timestamp',
+ dataIndex: 'last-run-endtime',
+ },
+ {
+ text: gettext('Duration'),
+ dataIndex: 'duration',
+ width: 60,
+ renderer: Proxmox.Utils.render_duration,
+ },
+ {
+ header: gettext('Next Run'),
+ sortable: true,
+ minWidth: 200,
+ renderer: 'render_next_run',
+ dataIndex: 'next-run',
+ },
+ {
+ header: gettext('Comment'),
+ hidden: true,
+ sortable: true,
+ renderer: Ext.String.htmlEncode,
+ dataIndex: 'comment',
+ },
+ ],
+});
--
2.20.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [pbs-devel] [PATCH v2 proxmox-backup 09/15] ui: add verify job edit window
2020-10-07 9:03 [pbs-devel] [PATCH v2 proxmox-backup 00/15] add job based verify scheduling Hannes Laimer
` (7 preceding siblings ...)
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 08/15] ui: add verify job view Hannes Laimer
@ 2020-10-07 9:03 ` Hannes Laimer
2020-10-08 9:58 ` Dominik Csapak
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 10/15] ui: add task descriptions for the different types of verify(job, snapshot, group, ds) Hannes Laimer
` (5 subsequent siblings)
14 siblings, 1 reply; 20+ messages in thread
From: Hannes Laimer @ 2020-10-07 9:03 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
www/Makefile | 1 +
www/window/VerifyJobEdit.js | 89 +++++++++++++++++++++++++++++++++++++
2 files changed, 90 insertions(+)
create mode 100644 www/window/VerifyJobEdit.js
diff --git a/www/Makefile b/www/Makefile
index a6c46718..e04af930 100644
--- a/www/Makefile
+++ b/www/Makefile
@@ -20,6 +20,7 @@ JSSRC= \
config/DataStoreConfig.js \
window/UserEdit.js \
window/UserPassword.js \
+ window/VerifyJobEdit.js \
window/RemoteEdit.js \
window/SyncJobEdit.js \
window/ACLEdit.js \
diff --git a/www/window/VerifyJobEdit.js b/www/window/VerifyJobEdit.js
new file mode 100644
index 00000000..985d2ef6
--- /dev/null
+++ b/www/window/VerifyJobEdit.js
@@ -0,0 +1,89 @@
+Ext.define('PBS.window.VerifyJobEdit', {
+ extend: 'Proxmox.window.Edit',
+ alias: 'widget.pbsVerifyJobEdit',
+ mixins: ['Proxmox.Mixin.CBind'],
+
+ userid: undefined,
+
+ onlineHelp: 'verifyjobs',
+
+ isAdd: true,
+
+ subject: gettext('VerifyJob'),
+
+ fieldDefaults: { labelWidth: 120 },
+
+ cbindData: function(initialConfig) {
+ let me = this;
+
+ let baseurl = '/api2/extjs/config/verify';
+ let id = initialConfig.id;
+
+ me.isCreate = !id;
+ me.url = id ? `${baseurl}/${id}` : baseurl;
+ me.method = id ? 'PUT' : 'POST';
+ me.autoLoad = !!id;
+ return { };
+ },
+
+ items: {
+ xtype: 'inputpanel',
+ column1: [
+ {
+ fieldLabel: gettext('Verify Job ID'),
+ xtype: 'pmxDisplayEditField',
+ name: 'id',
+ renderer: Ext.htmlEncode,
+ allowBlank: false,
+ minLength: 4,
+ cbind: {
+ editable: '{isCreate}',
+ },
+ },
+ {
+ fieldLabel: gettext('Datastore'),
+ xtype: 'pbsDataStoreSelector',
+ allowBlank: false,
+ name: 'store',
+ },
+ {
+ xtype: 'proxmoxintegerfield',
+ fieldLabel: gettext('Days valid'),
+ minValue: 1,
+ value: '',
+ allowBlank: true,
+ name: 'outdated-after'
+ },
+ ],
+
+ column2: [
+ {
+ fieldLabel: gettext('Ignore verified'),
+ xtype: 'proxmoxcheckbox',
+ name: 'ignore-verified',
+ uncheckedValue: false,
+ value: true,
+ },
+ {
+ fieldLabel: gettext('Schedule'),
+ xtype: 'pbsCalendarEvent',
+ name: 'schedule',
+ emptyText: gettext('none'),
+ cbind: {
+ deleteEmpty: '{!isCreate}',
+ },
+ },
+ ],
+
+ columnB: [
+ {
+ fieldLabel: gettext('Comment'),
+ xtype: 'proxmoxtextfield',
+ name: 'comment',
+ cbind: {
+ deleteEmpty: '{!isCreate}',
+ },
+ },
+ ],
+ },
+});
--
2.20.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [pbs-devel] [PATCH v2 proxmox-backup 10/15] ui: add task descriptions for the different types of verify(job, snapshot, group, ds)
2020-10-07 9:03 [pbs-devel] [PATCH v2 proxmox-backup 00/15] add job based verify scheduling Hannes Laimer
` (8 preceding siblings ...)
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 09/15] ui: add verify job edit window Hannes Laimer
@ 2020-10-07 9:03 ` Hannes Laimer
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 11/15] remove verify_schedule field from DatastoreConfig Hannes Laimer
` (4 subsequent siblings)
14 siblings, 0 replies; 20+ messages in thread
From: Hannes Laimer @ 2020-10-07 9:03 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
www/Utils.js | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/www/Utils.js b/www/Utils.js
index 8ad332eb..221a2f2b 100644
--- a/www/Utils.js
+++ b/www/Utils.js
@@ -91,9 +91,12 @@ Ext.define('PBS.Utils', {
Proxmox.Utils.override_task_descriptions({
garbage_collection: ['Datastore', gettext('Garbage collect')],
sync: ['Datastore', gettext('Remote Sync')],
+ verify: ['Datastore', gettext('Verification')],
+ verify_group: ['Group', gettext('Verification')],
+ verify_snapshot: ['Snapshot', gettext('Verification')],
syncjob: [gettext('Sync Job'), gettext('Remote Sync')],
+ verifyjob: [gettext('Verify Job'), gettext('Scheduled Verification')],
prune: (type, id) => PBS.Utils.render_datastore_worker_id(id, gettext('Prune')),
- verify: (type, id) => PBS.Utils.render_datastore_worker_id(id, gettext('Verify')),
backup: (type, id) => PBS.Utils.render_datastore_worker_id(id, gettext('Backup')),
reader: (type, id) => PBS.Utils.render_datastore_worker_id(id, gettext('Read objects')),
logrotate: [gettext('Log'), gettext('Rotation')],
--
2.20.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [pbs-devel] [PATCH v2 proxmox-backup 11/15] remove verify_schedule field from DatastoreConfig
2020-10-07 9:03 [pbs-devel] [PATCH v2 proxmox-backup 00/15] add job based verify scheduling Hannes Laimer
` (9 preceding siblings ...)
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 10/15] ui: add task descriptions for the different types of verify(job, snapshot, group, ds) Hannes Laimer
@ 2020-10-07 9:03 ` Hannes Laimer
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 12/15] remove verify_schedule field from datastore config endpoint Hannes Laimer
` (3 subsequent siblings)
14 siblings, 0 replies; 20+ messages in thread
From: Hannes Laimer @ 2020-10-07 9:03 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
src/config/datastore.rs | 6 ------
1 file changed, 6 deletions(-)
diff --git a/src/config/datastore.rs b/src/config/datastore.rs
index 1f5b4f84..7e2ae573 100644
--- a/src/config/datastore.rs
+++ b/src/config/datastore.rs
@@ -44,10 +44,6 @@ pub const DIR_NAME_SCHEMA: Schema = StringSchema::new("Directory name").schema()
optional: true,
schema: PRUNE_SCHEDULE_SCHEMA,
},
- "verify-schedule": {
- optional: true,
- schema: VERIFY_SCHEDULE_SCHEMA,
- },
"keep-last": {
optional: true,
schema: PRUNE_SCHEMA_KEEP_LAST,
@@ -87,8 +83,6 @@ pub struct DataStoreConfig {
#[serde(skip_serializing_if="Option::is_none")]
pub prune_schedule: Option<String>,
#[serde(skip_serializing_if="Option::is_none")]
- pub verify_schedule: Option<String>,
- #[serde(skip_serializing_if="Option::is_none")]
pub keep_last: Option<u64>,
#[serde(skip_serializing_if="Option::is_none")]
pub keep_hourly: Option<u64>,
--
2.20.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [pbs-devel] [PATCH v2 proxmox-backup 12/15] remove verify_schedule field from datastore config endpoint
2020-10-07 9:03 [pbs-devel] [PATCH v2 proxmox-backup 00/15] add job based verify scheduling Hannes Laimer
` (10 preceding siblings ...)
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 11/15] remove verify_schedule field from DatastoreConfig Hannes Laimer
@ 2020-10-07 9:03 ` Hannes Laimer
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 13/15] remove verify-schedule field from DataStoreEdit and DataStoreConfig Hannes Laimer
` (2 subsequent siblings)
14 siblings, 0 replies; 20+ messages in thread
From: Hannes Laimer @ 2020-10-07 9:03 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
src/api2/config/datastore.rs | 24 ------------------------
1 file changed, 24 deletions(-)
diff --git a/src/api2/config/datastore.rs b/src/api2/config/datastore.rs
index 07ca4ab8..767db1ea 100644
--- a/src/api2/config/datastore.rs
+++ b/src/api2/config/datastore.rs
@@ -75,10 +75,6 @@ pub fn list_datastores(
optional: true,
schema: PRUNE_SCHEDULE_SCHEMA,
},
- "verify-schedule": {
- optional: true,
- schema: VERIFY_SCHEDULE_SCHEMA,
- },
"keep-last": {
optional: true,
schema: PRUNE_SCHEMA_KEEP_LAST,
@@ -133,7 +129,6 @@ pub fn create_datastore(param: Value) -> Result<(), Error> {
crate::config::jobstate::create_state_file("prune", &datastore.name)?;
crate::config::jobstate::create_state_file("garbage_collection", &datastore.name)?;
- crate::config::jobstate::create_state_file("verify", &datastore.name)?;
Ok(())
}
@@ -179,8 +174,6 @@ pub enum DeletableProperty {
gc_schedule,
/// Delete the prune job schedule.
prune_schedule,
- /// Delete the verify schedule property
- verify_schedule,
/// Delete the keep-last property
keep_last,
/// Delete the keep-hourly property
@@ -214,10 +207,6 @@ pub enum DeletableProperty {
optional: true,
schema: PRUNE_SCHEDULE_SCHEMA,
},
- "verify-schedule": {
- optional: true,
- schema: VERIFY_SCHEDULE_SCHEMA,
- },
"keep-last": {
optional: true,
schema: PRUNE_SCHEMA_KEEP_LAST,
@@ -266,7 +255,6 @@ pub fn update_datastore(
comment: Option<String>,
gc_schedule: Option<String>,
prune_schedule: Option<String>,
- verify_schedule: Option<String>,
keep_last: Option<u64>,
keep_hourly: Option<u64>,
keep_daily: Option<u64>,
@@ -295,7 +283,6 @@ pub fn update_datastore(
DeletableProperty::comment => { data.comment = None; },
DeletableProperty::gc_schedule => { data.gc_schedule = None; },
DeletableProperty::prune_schedule => { data.prune_schedule = None; },
- DeletableProperty::verify_schedule => { data.verify_schedule = None; },
DeletableProperty::keep_last => { data.keep_last = None; },
DeletableProperty::keep_hourly => { data.keep_hourly = None; },
DeletableProperty::keep_daily => { data.keep_daily = None; },
@@ -327,12 +314,6 @@ pub fn update_datastore(
data.prune_schedule = prune_schedule;
}
- let mut verify_schedule_changed = false;
- if verify_schedule.is_some() {
- verify_schedule_changed = data.verify_schedule != verify_schedule;
- data.verify_schedule = verify_schedule;
- }
-
if keep_last.is_some() { data.keep_last = keep_last; }
if keep_hourly.is_some() { data.keep_hourly = keep_hourly; }
if keep_daily.is_some() { data.keep_daily = keep_daily; }
@@ -354,10 +335,6 @@ pub fn update_datastore(
crate::config::jobstate::create_state_file("prune", &name)?;
}
- if verify_schedule_changed {
- crate::config::jobstate::create_state_file("verify", &name)?;
- }
-
Ok(())
}
@@ -400,7 +377,6 @@ pub fn delete_datastore(name: String, digest: Option<String>) -> Result<(), Erro
// ignore errors
let _ = crate::config::jobstate::remove_state_file("prune", &name);
let _ = crate::config::jobstate::remove_state_file("garbage_collection", &name);
- let _ = crate::config::jobstate::remove_state_file("verify", &name);
Ok(())
}
--
2.20.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [pbs-devel] [PATCH v2 proxmox-backup 13/15] remove verify-schedule field from DataStoreEdit and DataStoreConfig
2020-10-07 9:03 [pbs-devel] [PATCH v2 proxmox-backup 00/15] add job based verify scheduling Hannes Laimer
` (11 preceding siblings ...)
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 12/15] remove verify_schedule field from datastore config endpoint Hannes Laimer
@ 2020-10-07 9:03 ` Hannes Laimer
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 14/15] remove old verification scheduling from proxmox-backup-proxy.rs Hannes Laimer
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 15/15] verify job: log number of planned verifications Hannes Laimer
14 siblings, 0 replies; 20+ messages in thread
From: Hannes Laimer @ 2020-10-07 9:03 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
www/config/DataStoreConfig.js | 2 +-
www/window/DataStoreEdit.js | 9 ---------
2 files changed, 1 insertion(+), 10 deletions(-)
diff --git a/www/config/DataStoreConfig.js b/www/config/DataStoreConfig.js
index b7e78e2f..440feea5 100644
--- a/www/config/DataStoreConfig.js
+++ b/www/config/DataStoreConfig.js
@@ -12,7 +12,7 @@ Ext.define('pbs-data-store-config', {
extend: 'Ext.data.Model',
fields: [
'name', 'path', 'comment', 'gc-schedule', 'prune-schedule',
- 'verify-schedule', 'keep-last', 'keep-hourly', 'keep-daily',
+ 'keep-last', 'keep-hourly', 'keep-daily',
'keep-weekly', 'keep-monthly', 'keep-yearly',
],
proxy: {
diff --git a/www/window/DataStoreEdit.js b/www/window/DataStoreEdit.js
index f565cee5..3cf218f1 100644
--- a/www/window/DataStoreEdit.js
+++ b/www/window/DataStoreEdit.js
@@ -74,15 +74,6 @@ Ext.define('PBS.DataStoreEdit', {
deleteEmpty: '{!isCreate}',
},
},
- {
- xtype: 'pbsCalendarEvent',
- name: 'verify-schedule',
- fieldLabel: gettext("Verify Schedule"),
- emptyText: gettext('none'),
- cbind: {
- deleteEmpty: '{!isCreate}',
- },
- },
],
columnB: [
{
--
2.20.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [pbs-devel] [PATCH v2 proxmox-backup 14/15] remove old verification scheduling from proxmox-backup-proxy.rs
2020-10-07 9:03 [pbs-devel] [PATCH v2 proxmox-backup 00/15] add job based verify scheduling Hannes Laimer
` (12 preceding siblings ...)
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 13/15] remove verify-schedule field from DataStoreEdit and DataStoreConfig Hannes Laimer
@ 2020-10-07 9:03 ` Hannes Laimer
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 15/15] verify job: log number of planned verifications Hannes Laimer
14 siblings, 0 replies; 20+ messages in thread
From: Hannes Laimer @ 2020-10-07 9:03 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
src/bin/proxmox-backup-proxy.rs | 115 --------------------------------
1 file changed, 115 deletions(-)
diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
index 79d5fe50..1c0d3857 100644
--- a/src/bin/proxmox-backup-proxy.rs
+++ b/src/bin/proxmox-backup-proxy.rs
@@ -197,7 +197,6 @@ async fn schedule_tasks() -> Result<(), Error> {
schedule_datastore_garbage_collection().await;
schedule_datastore_prune().await;
- schedule_datastore_verification().await;
schedule_datastore_sync_jobs().await;
schedule_datastore_verify_jobs().await;
schedule_task_log_rotate().await;
@@ -471,120 +470,6 @@ async fn schedule_datastore_prune() {
}
}
-async fn schedule_datastore_verification() {
- use proxmox_backup::backup::{DataStore, verify_all_backups};
- use proxmox_backup::server::{WorkerTask};
- use proxmox_backup::config::{
- jobstate::{self, Job},
- datastore::{self, DataStoreConfig}
- };
- use proxmox_backup::tools::systemd::time::{
- parse_calendar_event, compute_next_event};
-
- let config = match datastore::config() {
- Err(err) => {
- eprintln!("unable to read datastore config - {}", err);
- return;
- }
- Ok((config, _digest)) => config,
- };
-
- for (store, (_, store_config)) in config.sections {
- let datastore = match DataStore::lookup_datastore(&store) {
- Ok(datastore) => datastore,
- Err(err) => {
- eprintln!("lookup_datastore failed - {}", err);
- continue;
- }
- };
-
- let store_config: DataStoreConfig = match serde_json::from_value(store_config) {
- Ok(c) => c,
- Err(err) => {
- eprintln!("datastore config from_value failed - {}", err);
- continue;
- }
- };
-
- let event_str = match store_config.verify_schedule {
- Some(event_str) => event_str,
- None => continue,
- };
-
- let event = match parse_calendar_event(&event_str) {
- Ok(event) => event,
- Err(err) => {
- eprintln!("unable to parse schedule '{}' - {}", event_str, err);
- continue;
- }
- };
-
- let worker_type = "verify";
-
- let last = match jobstate::last_run_time(worker_type, &store) {
- Ok(time) => time,
- Err(err) => {
- eprintln!("could not get last run time of {} {}: {}", worker_type, store, err);
- continue;
- }
- };
-
- let next = match compute_next_event(&event, last, false) {
- Ok(Some(next)) => next,
- Ok(None) => continue,
- Err(err) => {
- eprintln!("compute_next_event for '{}' failed - {}", event_str, err);
- continue;
- }
- };
-
- let now = proxmox::tools::time::epoch_i64();
-
- if next > now { continue; }
-
- let mut job = match Job::new(worker_type, &store) {
- Ok(job) => job,
- Err(_) => continue, // could not get lock
- };
-
- let worker_id = store.clone();
- let store2 = store.clone();
- if let Err(err) = WorkerTask::new_thread(
- worker_type,
- Some(worker_id),
- Userid::backup_userid().clone(),
- false,
- move |worker| {
- job.start(&worker.upid().to_string())?;
- worker.log(format!("starting verification on store {}", store2));
- worker.log(format!("task triggered by schedule '{}'", event_str));
- let result = try_block!({
- let failed_dirs = verify_all_backups(datastore, worker.clone())?;
- if failed_dirs.len() > 0 {
- worker.log("Failed to verify following snapshots:");
- for dir in failed_dirs {
- worker.log(format!("\t{}", dir));
- }
- Err(format_err!("verification failed - please check the log for details"))
- } else {
- Ok(())
- }
- });
-
- let status = worker.create_state(&result);
-
- if let Err(err) = job.finish(status) {
- eprintln!("could not finish job state for {}: {}", worker_type, err);
- }
-
- result
- },
- ) {
- eprintln!("unable to start verification on store {} - {}", store, err);
- }
- }
-}
-
async fn schedule_datastore_sync_jobs() {
use proxmox_backup::{
--
2.20.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [pbs-devel] [PATCH v2 proxmox-backup 15/15] verify job: log number of planned verifications
2020-10-07 9:03 [pbs-devel] [PATCH v2 proxmox-backup 00/15] add job based verify scheduling Hannes Laimer
` (13 preceding siblings ...)
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 14/15] remove old verification scheduling from proxmox-backup-proxy.rs Hannes Laimer
@ 2020-10-07 9:03 ` Hannes Laimer
14 siblings, 0 replies; 20+ messages in thread
From: Hannes Laimer @ 2020-10-07 9:03 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
src/backup/verify.rs | 1 +
1 file changed, 1 insertion(+)
diff --git a/src/backup/verify.rs b/src/backup/verify.rs
index 517ecbb2..fac2677e 100644
--- a/src/backup/verify.rs
+++ b/src/backup/verify.rs
@@ -459,6 +459,7 @@ pub fn do_verification_job(
if let Some(event_str) = schedule {
worker.log(format!("task triggered by schedule '{}'", event_str));
}
+ worker.log(format!("verifying {} backups", backups_to_verify.len()));
let result = proxmox::try_block!({
let mut failed_dirs: Vec<String> = Vec::new();
for backup_info in backups_to_verify {
--
2.20.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [pbs-devel] [PATCH v2 proxmox-backup 02/15] add verify job config
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 02/15] add verify job config Hannes Laimer
@ 2020-10-08 7:50 ` Wolfgang Bumiller
2020-10-08 8:15 ` Dominik Csapak
0 siblings, 1 reply; 20+ messages in thread
From: Wolfgang Bumiller @ 2020-10-08 7:50 UTC (permalink / raw)
To: Hannes Laimer; +Cc: pbs-devel
On Wed, Oct 07, 2020 at 11:03:11AM +0200, Hannes Laimer wrote:
> Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> ---
> src/config.rs | 1 +
> src/config/verify.rs | 189 +++++++++++++++++++++++++++++++++++++++++++
> 2 files changed, 190 insertions(+)
> create mode 100644 src/config/verify.rs
>
> diff --git a/src/config.rs b/src/config.rs
> index c2ac6da1..ab7fc81a 100644
> --- a/src/config.rs
> +++ b/src/config.rs
> @@ -23,6 +23,7 @@ pub mod network;
> pub mod remote;
> pub mod sync;
> pub mod user;
> +pub mod verify;
>
> /// Check configuration directory permissions
> ///
> diff --git a/src/config/verify.rs b/src/config/verify.rs
> new file mode 100644
> index 00000000..7c92a16e
> --- /dev/null
> +++ b/src/config/verify.rs
> @@ -0,0 +1,189 @@
> +use anyhow::{Error};
> +use lazy_static::lazy_static;
> +use std::collections::HashMap;
> +use serde::{Serialize, Deserialize};
> +
> +use proxmox::api::{
> + api,
> + schema::*,
> + section_config::{
> + SectionConfig,
> + SectionConfigData,
> + SectionConfigPlugin,
> + }
> +};
> +
> +use proxmox::tools::{fs::replace_file, fs::CreateOptions};
> +
> +use crate::api2::types::*;
> +
> +lazy_static! {
> + static ref CONFIG: SectionConfig = init();
> +}
> +
> +
> +#[api(
> + properties: {
> + id: {
> + schema: JOB_ID_SCHEMA,
> + },
> + store: {
> + schema: DATASTORE_SCHEMA,
> + },
> + "ignore-verified": {
> + optional: true,
> + schema: IGNORE_VERIFIED_BACKUPS_SCHEMA,
> + },
> + "outdated-after": {
> + optional: true,
> + schema: VERIFICATION_OUTDATED_AFTER_SCHEMA,
> + },
> + comment: {
> + optional: true,
> + schema: SINGLE_LINE_COMMENT_SCHEMA,
> + },
> + schedule: {
> + optional: true,
> + schema: VERIFY_SCHEDULE_SCHEMA,
> + },
> + }
> +)]
> +#[serde(rename_all="kebab-case")]
> +#[derive(Serialize,Deserialize)]
> +/// Verify Job
> +pub struct VerifyJobConfig {
First things first - can we name this `VerificationJobConfig`? I'm not a
fan of using a verb here...
> + pub id: String,
> + pub store: String,
> + #[serde(skip_serializing_if="Option::is_none")]
> + pub ignore_verified: Option<bool>,
> + #[serde(skip_serializing_if="Option::is_none")]
> + pub outdated_after: Option<i64>,
> + #[serde(skip_serializing_if="Option::is_none")]
> + pub comment: Option<String>,
> + #[serde(skip_serializing_if="Option::is_none")]
> + pub schedule: Option<String>,
> +}
> +
> +// FIXME: generate duplicate schemas/structs from one listing?
Why though?
So my question there is, what is this supposed to actually represent?
It is a separate struct and it contains the keys from the other struct,
but how do they relate to each other? (And I mean apart from "we do this
in PVE, too", I do not consider that a good argument...)
Do you really need all of these fields in there or really just the id?
And if you want/need all the info as part of the state, what's stopping
you from simply using an actual struct member of type `VerifyJobConfig`
instead of flattening it out?
> +#[api(
> + properties: {
> + id: {
> + schema: JOB_ID_SCHEMA,
> + },
> + store: {
> + schema: DATASTORE_SCHEMA,
> + },
> + "ignore-verified": {
> + optional: true,
> + schema: IGNORE_VERIFIED_BACKUPS_SCHEMA,
> + },
> + "outdated-after": {
> + optional: true,
> + schema: VERIFICATION_OUTDATED_AFTER_SCHEMA,
> + },
> + comment: {
> + optional: true,
> + schema: SINGLE_LINE_COMMENT_SCHEMA,
> + },
> + schedule: {
> + optional: true,
> + schema: VERIFY_SCHEDULE_SCHEMA,
> + },
> + "next-run": {
> + description: "Estimated time of the next run (UNIX epoch).",
> + optional: true,
> + type: Integer,
> + },
> + "last-run-state": {
> + description: "Result of the last run.",
> + optional: true,
> + type: String,
> + },
> + "last-run-upid": {
> + description: "Task UPID of the last run.",
> + optional: true,
> + type: String,
> + },
> + "last-run-endtime": {
> + description: "Endtime of the last run.",
> + optional: true,
> + type: Integer,
> + },
> + }
> +)]
> +#[serde(rename_all="kebab-case")]
> +#[derive(Serialize,Deserialize)]
> +/// Status of Verify Job
> +pub struct VerifyJobStatus {
> + pub id: String,
> + pub store: String,
> + #[serde(skip_serializing_if="Option::is_none")]
> + pub ignore_verified: Option<bool>,
> + #[serde(skip_serializing_if="Option::is_none")]
> + pub outdated_after: Option<i64>,
> + #[serde(skip_serializing_if="Option::is_none")]
> + pub comment: Option<String>,
> + #[serde(skip_serializing_if="Option::is_none")]
> + pub schedule: Option<String>,
> + #[serde(skip_serializing_if="Option::is_none")]
> + pub next_run: Option<i64>,
> + #[serde(skip_serializing_if="Option::is_none")]
> + pub last_run_state: Option<String>,
> + #[serde(skip_serializing_if="Option::is_none")]
> + pub last_run_upid: Option<String>,
> + #[serde(skip_serializing_if="Option::is_none")]
> + pub last_run_endtime: Option<i64>,
> +}
> +
> +
> +fn init() -> SectionConfig {
> + let obj_schema = match VerifyJobConfig::API_SCHEMA {
> + Schema::Object(ref obj_schema) => obj_schema,
> + _ => unreachable!(),
> + };
> +
> + let plugin = SectionConfigPlugin::new("verify".to_string(), Some(String::from("id")), obj_schema);
> + let mut config = SectionConfig::new(&JOB_ID_SCHEMA);
> + config.register_plugin(plugin);
> +
> + config
> +}
> +
> +pub const VERIFY_CFG_FILENAME: &str = "/etc/proxmox-backup/verify.cfg";
> +pub const VERIFY_CFG_LOCKFILE: &str = "/etc/proxmox-backup/.verify.lck";
> +
> +pub fn config() -> Result<(SectionConfigData, [u8;32]), Error> {
> +
> + let content = proxmox::tools::fs::file_read_optional_string(VERIFY_CFG_FILENAME)?;
> + let content = content.unwrap_or(String::from(""));
Use `unwrap_or_else(String::new)`.
> +
> + let digest = openssl::sha::sha256(content.as_bytes());
> + let data = CONFIG.parse(VERIFY_CFG_FILENAME, &content)?;
> + Ok((data, digest))
> +}
> +
> +pub fn save_config(config: &SectionConfigData) -> Result<(), Error> {
> + let raw = CONFIG.write(VERIFY_CFG_FILENAME, &config)?;
> +
> + let backup_user = crate::backup::backup_user()?;
> + let mode = nix::sys::stat::Mode::from_bits_truncate(0o0640);
> + // set the correct owner/group/permissions while saving file
> + // owner(rw) = root, group(r)= backup
> +
> + let options = CreateOptions::new()
> + .perm(mode)
> + .owner(nix::unistd::ROOT)
> + .group(backup_user.gid);
> +
> + replace_file(VERIFY_CFG_FILENAME, raw.as_bytes(), options)?;
> +
> + Ok(())
> +}
> +
> +// shell completion helper
> +pub fn complete_verify_job_id(_arg: &str, _param: &HashMap<String, String>) -> Vec<String> {
> + match config() {
> + Ok((data, _digest)) => data.sections.iter().map(|(id, _)| id.to_string()).collect(),
> + Err(_) => return vec![],
> + }
> +}
> \ No newline at end of file
> --
> 2.20.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [pbs-devel] [PATCH v2 proxmox-backup 02/15] add verify job config
2020-10-08 7:50 ` Wolfgang Bumiller
@ 2020-10-08 8:15 ` Dominik Csapak
0 siblings, 0 replies; 20+ messages in thread
From: Dominik Csapak @ 2020-10-08 8:15 UTC (permalink / raw)
To: pbs-devel
On 10/8/20 9:50 AM, Wolfgang Bumiller wrote:
> On Wed, Oct 07, 2020 at 11:03:11AM +0200, Hannes Laimer wrote:
>> Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
>> ---
>> src/config.rs | 1 +
>> src/config/verify.rs | 189 +++++++++++++++++++++++++++++++++++++++++++
>> 2 files changed, 190 insertions(+)
>> create mode 100644 src/config/verify.rs
>>
>> diff --git a/src/config.rs b/src/config.rs
>> index c2ac6da1..ab7fc81a 100644
>> --- a/src/config.rs
>> +++ b/src/config.rs
>> @@ -23,6 +23,7 @@ pub mod network;
>> pub mod remote;
>> pub mod sync;
>> pub mod user;
>> +pub mod verify;
>>
>> /// Check configuration directory permissions
>> ///
>> diff --git a/src/config/verify.rs b/src/config/verify.rs
>> new file mode 100644
>> index 00000000..7c92a16e
>> --- /dev/null
>> +++ b/src/config/verify.rs
>> @@ -0,0 +1,189 @@
>> +use anyhow::{Error};
>> +use lazy_static::lazy_static;
>> +use std::collections::HashMap;
>> +use serde::{Serialize, Deserialize};
>> +
>> +use proxmox::api::{
>> + api,
>> + schema::*,
>> + section_config::{
>> + SectionConfig,
>> + SectionConfigData,
>> + SectionConfigPlugin,
>> + }
>> +};
>> +
>> +use proxmox::tools::{fs::replace_file, fs::CreateOptions};
>> +
>> +use crate::api2::types::*;
>> +
>> +lazy_static! {
>> + static ref CONFIG: SectionConfig = init();
>> +}
>> +
>> +
>> +#[api(
>> + properties: {
>> + id: {
>> + schema: JOB_ID_SCHEMA,
>> + },
>> + store: {
>> + schema: DATASTORE_SCHEMA,
>> + },
>> + "ignore-verified": {
>> + optional: true,
>> + schema: IGNORE_VERIFIED_BACKUPS_SCHEMA,
>> + },
>> + "outdated-after": {
>> + optional: true,
>> + schema: VERIFICATION_OUTDATED_AFTER_SCHEMA,
>> + },
>> + comment: {
>> + optional: true,
>> + schema: SINGLE_LINE_COMMENT_SCHEMA,
>> + },
>> + schedule: {
>> + optional: true,
>> + schema: VERIFY_SCHEDULE_SCHEMA,
>> + },
>> + }
>> +)]
>> +#[serde(rename_all="kebab-case")]
>> +#[derive(Serialize,Deserialize)]
>> +/// Verify Job
>> +pub struct VerifyJobConfig {
>
> First things first - can we name this `VerificationJobConfig`? I'm not a
> fan of using a verb here...
>
>> + pub id: String,
>> + pub store: String,
>> + #[serde(skip_serializing_if="Option::is_none")]
>> + pub ignore_verified: Option<bool>,
>> + #[serde(skip_serializing_if="Option::is_none")]
>> + pub outdated_after: Option<i64>,
>> + #[serde(skip_serializing_if="Option::is_none")]
>> + pub comment: Option<String>,
>> + #[serde(skip_serializing_if="Option::is_none")]
>> + pub schedule: Option<String>,
>> +}
>> +
>> +// FIXME: generate duplicate schemas/structs from one listing?
>
> Why though?
>
> So my question there is, what is this supposed to actually represent?
>
> It is a separate struct and it contains the keys from the other struct,
> but how do they relate to each other? (And I mean apart from "we do this
> in PVE, too", I do not consider that a good argument...)
>
> Do you really need all of these fields in there or really just the id?
>
> And if you want/need all the info as part of the state, what's stopping
> you from simply using an actual struct member of type `VerifyJobConfig`
> instead of flattening it out?
>
fyi, hannes probably copied it from our sync jobs
the rationale here is that we need a list of all those properties below
in a flat array (so no nested object) for easy use in the gui
and we wanted to have a strict type (Vec<VerifyJobStatus>) instead of
Vec<Value> (or simply Value) to get some compile time guarantees
but the additional fields do not belong in the config so we needed
a second struct
>> +#[api(
>> + properties: {
>> + id: {
>> + schema: JOB_ID_SCHEMA,
>> + },
>> + store: {
>> + schema: DATASTORE_SCHEMA,
>> + },
>> + "ignore-verified": {
>> + optional: true,
>> + schema: IGNORE_VERIFIED_BACKUPS_SCHEMA,
>> + },
>> + "outdated-after": {
>> + optional: true,
>> + schema: VERIFICATION_OUTDATED_AFTER_SCHEMA,
>> + },
>> + comment: {
>> + optional: true,
>> + schema: SINGLE_LINE_COMMENT_SCHEMA,
>> + },
>> + schedule: {
>> + optional: true,
>> + schema: VERIFY_SCHEDULE_SCHEMA,
>> + },
>> + "next-run": {
>> + description: "Estimated time of the next run (UNIX epoch).",
>> + optional: true,
>> + type: Integer,
>> + },
>> + "last-run-state": {
>> + description: "Result of the last run.",
>> + optional: true,
>> + type: String,
>> + },
>> + "last-run-upid": {
>> + description: "Task UPID of the last run.",
>> + optional: true,
>> + type: String,
>> + },
>> + "last-run-endtime": {
>> + description: "Endtime of the last run.",
>> + optional: true,
>> + type: Integer,
>> + },
>> + }
>> +)]
>> +#[serde(rename_all="kebab-case")]
>> +#[derive(Serialize,Deserialize)]
>> +/// Status of Verify Job
>> +pub struct VerifyJobStatus {
>> + pub id: String,
>> + pub store: String,
>> + #[serde(skip_serializing_if="Option::is_none")]
>> + pub ignore_verified: Option<bool>,
>> + #[serde(skip_serializing_if="Option::is_none")]
>> + pub outdated_after: Option<i64>,
>> + #[serde(skip_serializing_if="Option::is_none")]
>> + pub comment: Option<String>,
>> + #[serde(skip_serializing_if="Option::is_none")]
>> + pub schedule: Option<String>,
>> + #[serde(skip_serializing_if="Option::is_none")]
>> + pub next_run: Option<i64>,
>> + #[serde(skip_serializing_if="Option::is_none")]
>> + pub last_run_state: Option<String>,
>> + #[serde(skip_serializing_if="Option::is_none")]
>> + pub last_run_upid: Option<String>,
>> + #[serde(skip_serializing_if="Option::is_none")]
>> + pub last_run_endtime: Option<i64>,
>> +}
>> +
>> +
>> +fn init() -> SectionConfig {
>> + let obj_schema = match VerifyJobConfig::API_SCHEMA {
>> + Schema::Object(ref obj_schema) => obj_schema,
>> + _ => unreachable!(),
>> + };
>> +
>> + let plugin = SectionConfigPlugin::new("verify".to_string(), Some(String::from("id")), obj_schema);
>> + let mut config = SectionConfig::new(&JOB_ID_SCHEMA);
>> + config.register_plugin(plugin);
>> +
>> + config
>> +}
>> +
>> +pub const VERIFY_CFG_FILENAME: &str = "/etc/proxmox-backup/verify.cfg";
>> +pub const VERIFY_CFG_LOCKFILE: &str = "/etc/proxmox-backup/.verify.lck";
>> +
>> +pub fn config() -> Result<(SectionConfigData, [u8;32]), Error> {
>> +
>> + let content = proxmox::tools::fs::file_read_optional_string(VERIFY_CFG_FILENAME)?;
>> + let content = content.unwrap_or(String::from(""));
>
> Use `unwrap_or_else(String::new)`.
>
>> +
>> + let digest = openssl::sha::sha256(content.as_bytes());
>> + let data = CONFIG.parse(VERIFY_CFG_FILENAME, &content)?;
>> + Ok((data, digest))
>> +}
>> +
>> +pub fn save_config(config: &SectionConfigData) -> Result<(), Error> {
>> + let raw = CONFIG.write(VERIFY_CFG_FILENAME, &config)?;
>> +
>> + let backup_user = crate::backup::backup_user()?;
>> + let mode = nix::sys::stat::Mode::from_bits_truncate(0o0640);
>> + // set the correct owner/group/permissions while saving file
>> + // owner(rw) = root, group(r)= backup
>> +
>> + let options = CreateOptions::new()
>> + .perm(mode)
>> + .owner(nix::unistd::ROOT)
>> + .group(backup_user.gid);
>> +
>> + replace_file(VERIFY_CFG_FILENAME, raw.as_bytes(), options)?;
>> +
>> + Ok(())
>> +}
>> +
>> +// shell completion helper
>> +pub fn complete_verify_job_id(_arg: &str, _param: &HashMap<String, String>) -> Vec<String> {
>> + match config() {
>> + Ok((data, _digest)) => data.sections.iter().map(|(id, _)| id.to_string()).collect(),
>> + Err(_) => return vec![],
>> + }
>> +}
>> \ No newline at end of file
>> --
>> 2.20.1
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [pbs-devel] [PATCH v2 proxmox-backup 03/15] api2: add verify job config endpoint
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 03/15] api2: add verify job config endpoint Hannes Laimer
@ 2020-10-08 8:22 ` Wolfgang Bumiller
0 siblings, 0 replies; 20+ messages in thread
From: Wolfgang Bumiller @ 2020-10-08 8:22 UTC (permalink / raw)
To: Hannes Laimer; +Cc: pbs-devel
On Wed, Oct 07, 2020 at 11:03:12AM +0200, Hannes Laimer wrote:
> Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> ---
> src/api2/config.rs | 2 +
> src/api2/config/verify.rs | 275 ++++++++++++++++++++++++++++++++++++++
> 2 files changed, 277 insertions(+)
> create mode 100644 src/api2/config/verify.rs
>
> diff --git a/src/api2/config.rs b/src/api2/config.rs
> index be7397c8..7a5129c7 100644
> --- a/src/api2/config.rs
> +++ b/src/api2/config.rs
> @@ -4,11 +4,13 @@ use proxmox::list_subdirs_api_method;
> pub mod datastore;
> pub mod remote;
> pub mod sync;
> +pub mod verify;
>
> const SUBDIRS: SubdirMap = &[
> ("datastore", &datastore::ROUTER),
> ("remote", &remote::ROUTER),
> ("sync", &sync::ROUTER),
> + ("verify", &verify::ROUTER)
> ];
>
> pub const ROUTER: Router = Router::new()
> diff --git a/src/api2/config/verify.rs b/src/api2/config/verify.rs
> new file mode 100644
> index 00000000..9897aaf9
> --- /dev/null
> +++ b/src/api2/config/verify.rs
> @@ -0,0 +1,275 @@
> +use anyhow::{bail, Error};
> +use serde_json::Value;
> +use ::serde::{Deserialize, Serialize};
> +
> +use proxmox::api::{api, Router, RpcEnvironment};
> +use proxmox::tools::fs::open_file_locked;
> +
> +use crate::api2::types::*;
> +use crate::config::verify::{self, VerifyJobConfig};
> +
> +#[api(
> + input: {
> + properties: {},
> + },
> + returns: {
> + description: "List configured jobs.",
> + type: Array,
> + items: { type: verify::VerifyJobConfig },
> + },
> +)]
> +/// List all verify jobs
> +pub fn list_verify_jobs(
> + _param: Value,
> + mut rpcenv: &mut dyn RpcEnvironment,
> +) -> Result<Vec<VerifyJobConfig>, Error> {
> +
> + let (config, digest) = verify::config()?;
> +
> + let list = config.convert_to_typed_array("verify")?;
> +
> + rpcenv["digest"] = proxmox::tools::digest_to_hex(&digest).into();
> +
> + Ok(list)
> +}
> +
> +
> +#[api(
> + protected: true,
> + input: {
> + properties: {
> + id: {
> + schema: JOB_ID_SCHEMA,
> + },
> + store: {
> + schema: DATASTORE_SCHEMA,
> + },
> + "ignore-verified": {
> + optional: true,
> + schema: IGNORE_VERIFIED_BACKUPS_SCHEMA,
> + },
> + "outdated-after": {
> + optional: true,
> + schema: VERIFICATION_OUTDATED_AFTER_SCHEMA,
> + },
> + comment: {
> + optional: true,
> + schema: SINGLE_LINE_COMMENT_SCHEMA,
> + },
> + schedule: {
> + optional: true,
> + schema: VERIFY_SCHEDULE_SCHEMA,
> + },
> + }
> + }
> +)]
> +/// Create a new verify job.
> +pub fn create_verify_job(param: Value) -> Result<(), Error> {
> +
> + let _lock = open_file_locked(verify::VERIFY_CFG_LOCKFILE, std::time::Duration::new(10, 0), true)?;
> +
> + let verify_job: verify::VerifyJobConfig = serde_json::from_value(param.clone())?;
> +
> + let (mut config, _digest) = verify::config()?;
> +
> + if let Some(_) = config.sections.get(&verify_job.id) {
> + bail!("job '{}' already exists.", verify_job.id);
> + }
> +
> + config.set_data(&verify_job.id, "verify", &verify_job)?;
> +
> + verify::save_config(&config)?;
> +
> + crate::config::jobstate::create_state_file("verifyjob", &verify_job.id)?;
> +
> + Ok(())
> +}
> +
> +#[api(
> + input: {
> + properties: {
> + id: {
> + schema: JOB_ID_SCHEMA,
> + },
> + },
> + },
> + returns: {
> + description: "The verify job configuration.",
> + type: verify::VerifyJobConfig,
> + },
> +)]
> +/// Read a verify job configuration.
> +pub fn read_verify_job(
> + id: String,
> + mut rpcenv: &mut dyn RpcEnvironment,
> +) -> Result<VerifyJobConfig, Error> {
> + let (config, digest) = verify::config()?;
> +
> + let verify_job = config.lookup("verify", &id)?;
> + rpcenv["digest"] = proxmox::tools::digest_to_hex(&digest).into();
> +
> + Ok(verify_job)
> +}
> +
> +#[api()]
> +#[derive(Serialize, Deserialize)]
> +#[serde(rename_all="kebab-case")]
> +#[allow(non_camel_case_types)]
You already `rename-all`, please drop the non-camel-case-types and just
use standard style.
> +/// Deletable property name
> +pub enum DeletableProperty {
> + /// Delete ignore verified
> + ignore_verified,
> + /// Delete the comment property.
> + comment,
> + /// Delete the job schedule.
> + schedule,
> + /// Delete outdated_after.
> + outdated_after
> +}
> +
> +#[api(
> + protected: true,
> + input: {
> + properties: {
> + id: {
> + schema: JOB_ID_SCHEMA,
> + },
> + store: {
> + optional: true,
> + schema: DATASTORE_SCHEMA,
> + },
> + "ignore-verified": {
> + optional: true,
> + schema: IGNORE_VERIFIED_BACKUPS_SCHEMA,
> + },
> + "outdated-after": {
> + optional: true,
> + schema: VERIFICATION_OUTDATED_AFTER_SCHEMA,
> + },
> + comment: {
> + optional: true,
> + schema: SINGLE_LINE_COMMENT_SCHEMA,
> + },
> + schedule: {
> + optional: true,
> + schema: VERIFY_SCHEDULE_SCHEMA,
> + },
^ This is the part that needs actual fixing in the schema, because this
is where all the properties become optional. (That wouldn't striclty be
necessary, though, I mean most of the time we have the complete data
available already anyway before doing the call...)
But before this can be fixed in the `#[api]` macro it needs to be fixed
in the schema code itself.
> + delete: {
> + description: "List of properties to delete.",
> + type: Array,
> + optional: true,
> + items: {
> + type: DeletableProperty,
> + }
> + },
> + digest: {
> + optional: true,
> + schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
> + },
> + },
> + },
> +)]
> +/// Update verify job config.
> +pub fn update_verify_job(
> + id: String,
> + store: Option<String>,
> + ignore_verified: Option<bool>,
> + outdated_after: Option<i64>,
> + comment: Option<String>,
> + schedule: Option<String>,
> + delete: Option<Vec<DeletableProperty>>,
> + digest: Option<String>,
> +) -> Result<(), Error> {
> +
> + let _lock = open_file_locked(verify::VERIFY_CFG_LOCKFILE, std::time::Duration::new(10, 0), true)?;
> +
> + // pass/compare digest
> + let (mut config, expected_digest) = verify::config()?;
> +
> + if let Some(ref digest) = digest {
> + let digest = proxmox::tools::hex_to_digest(digest)?;
> + crate::tools::detect_modified_configuration_file(&digest, &expected_digest)?;
> + }
> +
> + let mut data: verify::VerifyJobConfig = config.lookup("verify", &id)?;
> +
> + if let Some(delete) = delete {
> + for delete_prop in delete {
> + match delete_prop {
> + DeletableProperty::ignore_verified => { data.ignore_verified = None; },
> + DeletableProperty::outdated_after => { data.outdated_after = None; },
> + DeletableProperty::comment => { data.comment = None; },
> + DeletableProperty::schedule => { data.schedule = None; },
> + }
> + }
> + }
> +
> + if let Some(comment) = comment {
> + let comment = comment.trim().to_string();
> + if comment.is_empty() {
> + data.comment = None;
> + } else {
> + data.comment = Some(comment);
> + }
> + }
> +
> + if let Some(store) = store { data.store = store; }
> +
> + if ignore_verified.is_some() { data.ignore_verified = ignore_verified; }
> + if outdated_after.is_some() { data.outdated_after = outdated_after; }
> + if schedule.is_some() { data.schedule = schedule; }
> +
> + config.set_data(&id, "verify", &data)?;
> +
> + verify::save_config(&config)?;
> +
> + Ok(())
> +}
> +
> +#[api(
> + protected: true,
> + input: {
> + properties: {
> + id: {
> + schema: JOB_ID_SCHEMA,
> + },
> + digest: {
> + optional: true,
> + schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
> + },
> + },
> + },
> +)]
> +/// Remove a verify job configuration
> +pub fn delete_verify_job(id: String, digest: Option<String>) -> Result<(), Error> {
> +
> + let _lock = open_file_locked(verify::VERIFY_CFG_LOCKFILE, std::time::Duration::new(10, 0), true)?;
> +
> + let (mut config, expected_digest) = verify::config()?;
> +
> + if let Some(ref digest) = digest {
> + let digest = proxmox::tools::hex_to_digest(digest)?;
> + crate::tools::detect_modified_configuration_file(&digest, &expected_digest)?;
> + }
> +
> + match config.sections.get(&id) {
> + Some(_) => { config.sections.remove(&id); },
> + None => bail!("job '{}' does not exist.", id),
> + }
> +
> + verify::save_config(&config)?;
> +
> + crate::config::jobstate::remove_state_file("verifyjob", &id)?;
> +
> + Ok(())
> +}
> +
> +const ITEM_ROUTER: Router = Router::new()
> + .get(&API_METHOD_READ_VERIFY_JOB)
> + .put(&API_METHOD_UPDATE_VERIFY_JOB)
> + .delete(&API_METHOD_DELETE_VERIFY_JOB);
> +
> +pub const ROUTER: Router = Router::new()
> + .get(&API_METHOD_LIST_VERIFY_JOBS)
> + .post(&API_METHOD_CREATE_VERIFY_JOB)
> + .match_all("id", &ITEM_ROUTER);
> \ No newline at end of file
> --
> 2.20.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [pbs-devel] [PATCH v2 proxmox-backup 09/15] ui: add verify job edit window
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 09/15] ui: add verify job edit window Hannes Laimer
@ 2020-10-08 9:58 ` Dominik Csapak
0 siblings, 0 replies; 20+ messages in thread
From: Dominik Csapak @ 2020-10-08 9:58 UTC (permalink / raw)
To: pbs-devel
one comment inline
On 10/7/20 11:03 AM, Hannes Laimer wrote:
> Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> ---
> www/Makefile | 1 +
> www/window/VerifyJobEdit.js | 89 +++++++++++++++++++++++++++++++++++++
> 2 files changed, 90 insertions(+)
> create mode 100644 www/window/VerifyJobEdit.js
>
> diff --git a/www/Makefile b/www/Makefile
> index a6c46718..e04af930 100644
> --- a/www/Makefile
> +++ b/www/Makefile
> @@ -20,6 +20,7 @@ JSSRC= \
> config/DataStoreConfig.js \
> window/UserEdit.js \
> window/UserPassword.js \
> + window/VerifyJobEdit.js \
> window/RemoteEdit.js \
> window/SyncJobEdit.js \
> window/ACLEdit.js \
> diff --git a/www/window/VerifyJobEdit.js b/www/window/VerifyJobEdit.js
> new file mode 100644
> index 00000000..985d2ef6
> --- /dev/null
> +++ b/www/window/VerifyJobEdit.js
> @@ -0,0 +1,89 @@
> +Ext.define('PBS.window.VerifyJobEdit', {
> + extend: 'Proxmox.window.Edit',
> + alias: 'widget.pbsVerifyJobEdit',
> + mixins: ['Proxmox.Mixin.CBind'],
> +
> + userid: undefined,
> +
> + onlineHelp: 'verifyjobs',
> +
> + isAdd: true,
> +
> + subject: gettext('VerifyJob'),
> +
> + fieldDefaults: { labelWidth: 120 },
> +
> + cbindData: function(initialConfig) {
> + let me = this;
> +
> + let baseurl = '/api2/extjs/config/verify';
> + let id = initialConfig.id;
> +
> + me.isCreate = !id;
> + me.url = id ? `${baseurl}/${id}` : baseurl;
> + me.method = id ? 'PUT' : 'POST';
> + me.autoLoad = !!id;
> + return { };
> + },
> +
> + items: {
> + xtype: 'inputpanel',
> + column1: [
> + {
> + fieldLabel: gettext('Verify Job ID'),
> + xtype: 'pmxDisplayEditField',
> + name: 'id',
> + renderer: Ext.htmlEncode,
> + allowBlank: false,
> + minLength: 4,
> + cbind: {
> + editable: '{isCreate}',
> + },
> + },
> + {
> + fieldLabel: gettext('Datastore'),
> + xtype: 'pbsDataStoreSelector',
> + allowBlank: false,
> + name: 'store',
> + },
> + {
> + xtype: 'proxmoxintegerfield',
> + fieldLabel: gettext('Days valid'),
> + minValue: 1,
> + value: '',
> + allowBlank: true,
> + name: 'outdated-after'
i would also add a
cbind: {
deleteEmpty: '{!isCreate}'
}
here (basically every field that is optional and can be empty should
have that)
> + },
> + ],
> +
> + column2: [
> + {
> + fieldLabel: gettext('Ignore verified'),
> + xtype: 'proxmoxcheckbox',
> + name: 'ignore-verified',
> + uncheckedValue: false,
> + value: true,
> + },
> + {
> + fieldLabel: gettext('Schedule'),
> + xtype: 'pbsCalendarEvent',
> + name: 'schedule',
> + emptyText: gettext('none'),
> + cbind: {
> + deleteEmpty: '{!isCreate}',
> + },
> + },
> + ],
> +
> + columnB: [
> + {
> + fieldLabel: gettext('Comment'),
> + xtype: 'proxmoxtextfield',
> + name: 'comment',
> + cbind: {
> + deleteEmpty: '{!isCreate}',
> + },
> + },
> + ],
> + },
> +});
>
^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2020-10-08 9:59 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-10-07 9:03 [pbs-devel] [PATCH v2 proxmox-backup 00/15] add job based verify scheduling Hannes Laimer
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 01/15] add two new schemas for verify jobs Hannes Laimer
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 02/15] add verify job config Hannes Laimer
2020-10-08 7:50 ` Wolfgang Bumiller
2020-10-08 8:15 ` Dominik Csapak
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 03/15] api2: add verify job config endpoint Hannes Laimer
2020-10-08 8:22 ` Wolfgang Bumiller
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 04/15] add do_verification_job function to verify.rs Hannes Laimer
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 05/15] api2: add verify job admin endpoint Hannes Laimer
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 06/15] add scheduling for verify jobs Hannes Laimer
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 07/15] set a diffrent worker_type based on what is going to be verified(snapshot, group, ds) Hannes Laimer
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 08/15] ui: add verify job view Hannes Laimer
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 09/15] ui: add verify job edit window Hannes Laimer
2020-10-08 9:58 ` Dominik Csapak
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 10/15] ui: add task descriptions for the different types of verify(job, snapshot, group, ds) Hannes Laimer
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 11/15] remove verify_schedule field from DatastoreConfig Hannes Laimer
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 12/15] remove verify_schedule field from datastore config endpoint Hannes Laimer
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 13/15] remove verify-schedule field from DataStoreEdit and DataStoreConfig Hannes Laimer
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 14/15] remove old verification scheduling from proxmox-backup-proxy.rs Hannes Laimer
2020-10-07 9:03 ` [pbs-devel] [PATCH v2 proxmox-backup 15/15] verify job: log number of planned verifications Hannes Laimer
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal