[PATCH datacenter-manager v2] Add notifications backend

[PATCH datacenter-manager v2] Add notifications backend

[Arthur Bied-Charreton]

Ideally, the whole router should be in proxmox-notify. However, after
discussing this quite extensively off-list with Lukas, we came to the
conclusion that doing so would require a large refactor that is
especially tricky given the fact that PVE would still need a different
entry point into proxmox-notify than PBS and PDM, since it defines its
router on the Perl side.

For this reason, factoring the routing logic out into proxmox-notify is
offset to a future where PVE also has a Rust-based router, and for now
it is just copied over from PBS (commit: fbf8d1b2) with the adaptations
described below.

The `proxmox_notify::context::Context` trait implementation is added to
PDM directly instead of creating a pdm.rs module in proxmox-notify to
avoid adding even more product-specific logic to it. It uses
`proxmox-access-control`, which is pulled in by PDM anyway, for getting
the user config instead of reading the file directly.

The `matcher-fields` and `matcher-field-values` endpoints currently
return empty vecs, they will be updated in a future commit introducing
actual notifications. The same applies for the notification templates.

The endpoints are made unprotected and the configuration files
read/writable by `www-data`, like for `remotes.cfg`.

Signed-off-by: Arthur Bied-Charreton <a.bied-charreton@proxmox.com>
---
 Cargo.toml                                    |   1 +
 Makefile                                      |   3 +-
 debian/proxmox-datacenter-manager.install     |   2 +
 defines.mk                                    |   1 +
 lib/pdm-config/Cargo.toml                     |   1 +
 lib/pdm-config/src/lib.rs                     |   1 +
 lib/pdm-config/src/notifications.rs           |  39 ++++
 server/Cargo.toml                             |   1 +
 server/src/api/config/mod.rs                  |   2 +
 server/src/api/config/notifications/gotify.rs | 185 +++++++++++++++++
 .../src/api/config/notifications/matchers.rs  | 165 ++++++++++++++++
 server/src/api/config/notifications/mod.rs    | 102 ++++++++++
 .../src/api/config/notifications/sendmail.rs  | 173 ++++++++++++++++
 server/src/api/config/notifications/smtp.rs   | 186 ++++++++++++++++++
 .../src/api/config/notifications/targets.rs   |  61 ++++++
 .../src/api/config/notifications/webhook.rs   | 170 ++++++++++++++++
 server/src/context.rs                         |   2 +
 server/src/lib.rs                             |   1 +
 server/src/notifications/mod.rs               | 115 +++++++++++
 templates/Makefile                            |  14 ++
 templates/default/test-body.txt.hbs           |   1 +
 templates/default/test-subject.txt.hbs        |   1 +
 22 files changed, 1226 insertions(+), 1 deletion(-)
 create mode 100644 lib/pdm-config/src/notifications.rs
 create mode 100644 server/src/api/config/notifications/gotify.rs
 create mode 100644 server/src/api/config/notifications/matchers.rs
 create mode 100644 server/src/api/config/notifications/mod.rs
 create mode 100644 server/src/api/config/notifications/sendmail.rs
 create mode 100644 server/src/api/config/notifications/smtp.rs
 create mode 100644 server/src/api/config/notifications/targets.rs
 create mode 100644 server/src/api/config/notifications/webhook.rs
 create mode 100644 server/src/notifications/mod.rs
 create mode 100644 templates/Makefile
 create mode 100644 templates/default/test-body.txt.hbs
 create mode 100644 templates/default/test-subject.txt.hbs

diff --git a/Cargo.toml b/Cargo.toml
index ec2aa3d..918e831 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -47,6 +47,7 @@ proxmox-ldap = { version = "1.1", features = ["sync"] }
 proxmox-lang = "1.1"
 proxmox-log = "1"
 proxmox-login = "1.0.2"
+proxmox-notify = "1"
 proxmox-rest-server = "1"
 # some use "cli", some use "cli" and "server", pbs-config uses nothing
 proxmox-router = { version = "3.0.0", default-features = false }
diff --git a/Makefile b/Makefile
index 8e1cd13..48327c9 100644
--- a/Makefile
+++ b/Makefile
@@ -79,6 +79,7 @@ install: $(COMPILED_BINS) $(SHELL_COMPLETION_FILES)
 	    install -m644 $(COMPLETION_DIR)/$(i) $(DESTDIR)$(ZSHCOMPDIR)/ ;)
 	make -C services install
 	$(MAKE) -C docs install
+	$(MAKE) -C templates install
 
 $(COMPILED_BINS) $(COMPILEDIR)/docgen &:
 	$(CARGO) build $(CARGO_BUILD_ARGS)
@@ -99,7 +100,7 @@ cargo-build:
 $(BUILDDIR):
 	rm -rf $@ $@.tmp
 	mkdir $@.tmp
-	cp -a debian/ server/ services/ cli/ lib/ docs/ ui/ defines.mk Makefile Cargo.toml $@.tmp
+	cp -a debian/ server/ services/ cli/ lib/ docs/ ui/ templates/ defines.mk Makefile Cargo.toml $@.tmp
 	echo "git clone git://git.proxmox.com/git/$(PACKAGE).git\\ngit checkout $$(git rev-parse HEAD)" \
 	    > $@.tmp/debian/SOURCE
 	mv $@.tmp $@
diff --git a/debian/proxmox-datacenter-manager.install b/debian/proxmox-datacenter-manager.install
index fad3f4a..fb2b1f1 100644
--- a/debian/proxmox-datacenter-manager.install
+++ b/debian/proxmox-datacenter-manager.install
@@ -20,3 +20,5 @@ usr/share/man/man5/remotes.cfg.5
 usr/share/man/man5/views.cfg.5
 usr/share/zsh/vendor-completions/_pdmAtoB
 usr/share/zsh/vendor-completions/_proxmox-datacenter-manager-admin
+usr/share/proxmox-datacenter-manager/templates/default/test-body.txt.hbs
+usr/share/proxmox-datacenter-manager/templates/default/test-subject.txt.hbs
diff --git a/defines.mk b/defines.mk
index 1b9c1a8..111923f 100644
--- a/defines.mk
+++ b/defines.mk
@@ -3,6 +3,7 @@ BINDIR = $(PREFIX)/bin
 SBINDIR = $(PREFIX)/sbin
 LIBDIR = $(PREFIX)/lib
 LIBEXECDIR = $(PREFIX)/libexec
+DATAROOTDIR = $(PREFIX)/share
 BASHCOMPDIR = $(PREFIX)/share/bash-completion/completions
 ZSHCOMPDIR = $(PREFIX)/share/zsh/vendor-completions
 MAN1DIR = $(PREFIX)/share/man/man1
diff --git a/lib/pdm-config/Cargo.toml b/lib/pdm-config/Cargo.toml
index d39c2ad..b6f3739 100644
--- a/lib/pdm-config/Cargo.toml
+++ b/lib/pdm-config/Cargo.toml
@@ -23,5 +23,6 @@ proxmox-shared-memory.workspace = true
 proxmox-simple-config.workspace = true
 proxmox-sys = { workspace = true, features = [ "acl", "crypt", "timer" ] }
 proxmox-acme-api.workspace = true
+proxmox-notify.workspace = true
 pdm-api-types.workspace = true
 pdm-buildcfg.workspace = true
diff --git a/lib/pdm-config/src/lib.rs b/lib/pdm-config/src/lib.rs
index 4c49054..03dc247 100644
--- a/lib/pdm-config/src/lib.rs
+++ b/lib/pdm-config/src/lib.rs
@@ -5,6 +5,7 @@ pub use pdm_buildcfg::{BACKUP_GROUP_NAME, BACKUP_USER_NAME};
 pub mod certificate_config;
 pub mod domains;
 pub mod node;
+pub mod notifications;
 pub mod remotes;
 pub mod setup;
 pub mod views;
diff --git a/lib/pdm-config/src/notifications.rs b/lib/pdm-config/src/notifications.rs
new file mode 100644
index 0000000..fd6ec79
--- /dev/null
+++ b/lib/pdm-config/src/notifications.rs
@@ -0,0 +1,39 @@
+use anyhow::Error;
+
+use proxmox_notify::Config;
+
+use pdm_buildcfg::configdir;
+use proxmox_product_config::{open_api_lockfile, replace_config, ApiLockGuard};
+use proxmox_sys::fs::file_read_optional_string;
+
+/// Configuration file location for notification targets/matchers.
+pub const NOTIFICATION_CONFIG_PATH: &str = configdir!("/notifications.cfg");
+
+/// Private configuration file location for secrets - only readable by `root`.
+pub const NOTIFICATION_PRIV_CONFIG_PATH: &str = configdir!("/notifications-priv.cfg");
+
+/// Lockfile to prevent concurrent write access.
+pub const NOTIFICATION_LOCK_FILE: &str = configdir!("/.notifications.lck");
+
+/// Get exclusive lock for `notifications.cfg`.
+pub fn lock_config() -> Result<ApiLockGuard, Error> {
+    open_api_lockfile(NOTIFICATION_LOCK_FILE, None, true)
+}
+
+/// Load notification config.
+pub fn config() -> Result<Config, Error> {
+    let content = file_read_optional_string(NOTIFICATION_CONFIG_PATH)?.unwrap_or_default();
+
+    let priv_content =
+        file_read_optional_string(NOTIFICATION_PRIV_CONFIG_PATH)?.unwrap_or_default();
+
+    Ok(Config::new(&content, &priv_content)?)
+}
+
+/// Save notification config.
+pub fn save_config(config: Config) -> Result<(), Error> {
+    let (cfg, priv_cfg) = config.write()?;
+    replace_config(NOTIFICATION_CONFIG_PATH, cfg.as_bytes())?;
+    replace_config(NOTIFICATION_PRIV_CONFIG_PATH, priv_cfg.as_bytes())?;
+    Ok(())
+}
diff --git a/server/Cargo.toml b/server/Cargo.toml
index 6969549..a4f7bbd 100644
--- a/server/Cargo.toml
+++ b/server/Cargo.toml
@@ -46,6 +46,7 @@ proxmox-lang.workspace = true
 proxmox-ldap.workspace = true
 proxmox-log.workspace = true
 proxmox-login.workspace = true
+proxmox-notify.workspace = true
 proxmox-openid.workspace = true
 proxmox-rest-server = { workspace = true, features = [ "templates" ] }
 proxmox-router = { workspace = true, features = [ "cli", "server"] }
diff --git a/server/src/api/config/mod.rs b/server/src/api/config/mod.rs
index 8f646c1..a465219 100644
--- a/server/src/api/config/mod.rs
+++ b/server/src/api/config/mod.rs
@@ -6,6 +6,7 @@ pub mod access;
 pub mod acme;
 pub mod certificate;
 pub mod notes;
+pub mod notifications;
 pub mod views;
 
 #[sortable]
@@ -14,6 +15,7 @@ const SUBDIRS: SubdirMap = &sorted!([
     ("acme", &acme::ROUTER),
     ("certificate", &certificate::ROUTER),
     ("notes", &notes::ROUTER),
+    ("notifications", &notifications::ROUTER),
     ("views", &views::ROUTER)
 ]);
 
diff --git a/server/src/api/config/notifications/gotify.rs b/server/src/api/config/notifications/gotify.rs
new file mode 100644
index 0000000..a281245
--- /dev/null
+++ b/server/src/api/config/notifications/gotify.rs
@@ -0,0 +1,185 @@
+use anyhow::Error;
+use serde_json::Value;
+
+use proxmox_notify::endpoints::gotify::{
+    DeleteableGotifyProperty, GotifyConfig, GotifyConfigUpdater, GotifyPrivateConfig,
+    GotifyPrivateConfigUpdater,
+};
+use proxmox_notify::schema::ENTITY_NAME_SCHEMA;
+use proxmox_router::{Permission, Router, RpcEnvironment};
+use proxmox_schema::api;
+
+use pbs_api_types::{PRIV_SYS_AUDIT, PRIV_SYS_MODIFY, PROXMOX_CONFIG_DIGEST_SCHEMA};
+
+#[api(
+    input: {
+        properties: {},
+    },
+    returns: {
+        description: "List of gotify endpoints.",
+        type: Array,
+        items: { type: GotifyConfig },
+    },
+    access: {
+        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false),
+    },
+)]
+/// List all gotify endpoints.
+pub fn list_endpoints(
+    _param: Value,
+    _rpcenv: &mut dyn RpcEnvironment,
+) -> Result<Vec<GotifyConfig>, Error> {
+    let config = pdm_config::notifications::config()?;
+
+    let endpoints = proxmox_notify::api::gotify::get_endpoints(&config)?;
+
+    Ok(endpoints)
+}
+
+#[api(
+    input: {
+        properties: {
+            name: {
+                schema: ENTITY_NAME_SCHEMA,
+            }
+        },
+    },
+    returns: { type: GotifyConfig },
+    access: {
+        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false),
+    },
+)]
+/// Get a gotify endpoint.
+pub fn get_endpoint(name: String, rpcenv: &mut dyn RpcEnvironment) -> Result<GotifyConfig, Error> {
+    let config = pdm_config::notifications::config()?;
+    let endpoint = proxmox_notify::api::gotify::get_endpoint(&config, &name)?;
+
+    rpcenv["digest"] = hex::encode(config.digest()).into();
+
+    Ok(endpoint)
+}
+
+#[api(
+    input: {
+        properties: {
+            endpoint: {
+                type: GotifyConfig,
+                flatten: true,
+            },
+            token: {
+                description: "Authentication token",
+            }
+        },
+    },
+    access: {
+        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
+    },
+)]
+/// Add a new gotify endpoint.
+pub fn add_endpoint(
+    endpoint: GotifyConfig,
+    token: String,
+    _rpcenv: &mut dyn RpcEnvironment,
+) -> Result<(), Error> {
+    let _lock = pdm_config::notifications::lock_config()?;
+    let mut config = pdm_config::notifications::config()?;
+    let private_endpoint_config = GotifyPrivateConfig {
+        name: endpoint.name.clone(),
+        token,
+    };
+
+    proxmox_notify::api::gotify::add_endpoint(&mut config, endpoint, private_endpoint_config)?;
+
+    pdm_config::notifications::save_config(config)?;
+    Ok(())
+}
+
+#[api(
+    input: {
+        properties: {
+            name: {
+                schema: ENTITY_NAME_SCHEMA,
+            },
+            updater: {
+                type: GotifyConfigUpdater,
+                flatten: true,
+            },
+            token: {
+                description: "Authentication token",
+                optional: true,
+            },
+            delete: {
+                description: "List of properties to delete.",
+                type: Array,
+                optional: true,
+                items: {
+                    type: DeleteableGotifyProperty,
+                }
+            },
+            digest: {
+                optional: true,
+                schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
+            },
+        },
+    },
+    access: {
+        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
+    },
+)]
+/// Update gotify endpoint.
+pub fn update_endpoint(
+    name: String,
+    updater: GotifyConfigUpdater,
+    token: Option<String>,
+    delete: Option<Vec<DeleteableGotifyProperty>>,
+    digest: Option<String>,
+    _rpcenv: &mut dyn RpcEnvironment,
+) -> Result<(), Error> {
+    let _lock = pdm_config::notifications::lock_config()?;
+    let mut config = pdm_config::notifications::config()?;
+    let digest = digest.map(hex::decode).transpose()?;
+
+    proxmox_notify::api::gotify::update_endpoint(
+        &mut config,
+        &name,
+        updater,
+        GotifyPrivateConfigUpdater { token },
+        delete.as_deref(),
+        digest.as_deref(),
+    )?;
+
+    pdm_config::notifications::save_config(config)?;
+    Ok(())
+}
+
+#[api(
+    input: {
+        properties: {
+            name: {
+                schema: ENTITY_NAME_SCHEMA,
+            }
+        },
+    },
+    access: {
+        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
+    },
+)]
+/// Delete gotify endpoint.
+pub fn delete_endpoint(name: String, _rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
+    let _lock = pdm_config::notifications::lock_config()?;
+    let mut config = pdm_config::notifications::config()?;
+    proxmox_notify::api::gotify::delete_gotify_endpoint(&mut config, &name)?;
+
+    pdm_config::notifications::save_config(config)?;
+    Ok(())
+}
+
+const ITEM_ROUTER: Router = Router::new()
+    .get(&API_METHOD_GET_ENDPOINT)
+    .put(&API_METHOD_UPDATE_ENDPOINT)
+    .delete(&API_METHOD_DELETE_ENDPOINT);
+
+pub const ROUTER: Router = Router::new()
+    .get(&API_METHOD_LIST_ENDPOINTS)
+    .post(&API_METHOD_ADD_ENDPOINT)
+    .match_all("name", &ITEM_ROUTER);
diff --git a/server/src/api/config/notifications/matchers.rs b/server/src/api/config/notifications/matchers.rs
new file mode 100644
index 0000000..9918305
--- /dev/null
+++ b/server/src/api/config/notifications/matchers.rs
@@ -0,0 +1,165 @@
+use anyhow::Error;
+use serde_json::Value;
+
+use proxmox_notify::matcher::{DeleteableMatcherProperty, MatcherConfig, MatcherConfigUpdater};
+use proxmox_notify::schema::ENTITY_NAME_SCHEMA;
+use proxmox_router::{Permission, Router, RpcEnvironment};
+use proxmox_schema::api;
+
+use pdm_api_types::{PRIV_SYS_AUDIT, PRIV_SYS_MODIFY, PROXMOX_CONFIG_DIGEST_SCHEMA};
+
+#[api(
+    input: {
+        properties: {},
+    },
+    returns: {
+        description: "List of matchers.",
+        type: Array,
+        items: { type: MatcherConfig },
+    },
+    access: {
+        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false),
+    },
+)]
+/// List all notification matchers.
+pub fn list_matchers(
+    _param: Value,
+    _rpcenv: &mut dyn RpcEnvironment,
+) -> Result<Vec<MatcherConfig>, Error> {
+    let config = pdm_config::notifications::config()?;
+
+    let matchers = proxmox_notify::api::matcher::get_matchers(&config)?;
+
+    Ok(matchers)
+}
+
+#[api(
+    input: {
+        properties: {
+            name: {
+                schema: ENTITY_NAME_SCHEMA,
+            }
+        },
+    },
+    returns: { type: MatcherConfig },
+    access: {
+        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false),
+    },
+)]
+/// Get a notification matcher.
+pub fn get_matcher(name: String, rpcenv: &mut dyn RpcEnvironment) -> Result<MatcherConfig, Error> {
+    let config = pdm_config::notifications::config()?;
+    let matcher = proxmox_notify::api::matcher::get_matcher(&config, &name)?;
+
+    rpcenv["digest"] = hex::encode(config.digest()).into();
+
+    Ok(matcher)
+}
+
+#[api(
+    input: {
+        properties: {
+            matcher: {
+                type: MatcherConfig,
+                flatten: true,
+            }
+        },
+    },
+    access: {
+        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
+    },
+)]
+/// Add a new notification matcher.
+pub fn add_matcher(matcher: MatcherConfig, _rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
+    let _lock = pdm_config::notifications::lock_config()?;
+    let mut config = pdm_config::notifications::config()?;
+
+    proxmox_notify::api::matcher::add_matcher(&mut config, matcher)?;
+
+    pdm_config::notifications::save_config(config)?;
+    Ok(())
+}
+
+#[api(
+    input: {
+        properties: {
+            name: {
+                schema: ENTITY_NAME_SCHEMA,
+            },
+            updater: {
+                type: MatcherConfigUpdater,
+                flatten: true,
+            },
+            delete: {
+                description: "List of properties to delete.",
+                type: Array,
+                optional: true,
+                items: {
+                    type: DeleteableMatcherProperty,
+                }
+            },
+            digest: {
+                optional: true,
+                schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
+            },
+        },
+    },
+    access: {
+        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
+    },
+)]
+/// Update notification matcher.
+pub fn update_matcher(
+    name: String,
+    updater: MatcherConfigUpdater,
+    delete: Option<Vec<DeleteableMatcherProperty>>,
+    digest: Option<String>,
+    _rpcenv: &mut dyn RpcEnvironment,
+) -> Result<(), Error> {
+    let _lock = pdm_config::notifications::lock_config()?;
+    let mut config = pdm_config::notifications::config()?;
+    let digest = digest.map(hex::decode).transpose()?;
+
+    proxmox_notify::api::matcher::update_matcher(
+        &mut config,
+        &name,
+        updater,
+        delete.as_deref(),
+        digest.as_deref(),
+    )?;
+
+    pdm_config::notifications::save_config(config)?;
+    Ok(())
+}
+
+#[api(
+    input: {
+        properties: {
+            name: {
+                schema: ENTITY_NAME_SCHEMA,
+            }
+        },
+    },
+    access: {
+        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
+    },
+)]
+/// Delete notification matcher.
+pub fn delete_matcher(name: String, _rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
+    let _lock = pdm_config::notifications::lock_config()?;
+    let mut config = pdm_config::notifications::config()?;
+    proxmox_notify::api::matcher::delete_matcher(&mut config, &name)?;
+
+    pdm_config::notifications::save_config(config)?;
+    Ok(())
+}
+
+const ITEM_ROUTER: Router = Router::new()
+    .get(&API_METHOD_GET_MATCHER)
+    .put(&API_METHOD_UPDATE_MATCHER)
+    .delete(&API_METHOD_DELETE_MATCHER);
+
+pub const ROUTER: Router = Router::new()
+    .get(&API_METHOD_LIST_MATCHERS)
+    .post(&API_METHOD_ADD_MATCHER)
+    .match_all("name", &ITEM_ROUTER);
diff --git a/server/src/api/config/notifications/mod.rs b/server/src/api/config/notifications/mod.rs
new file mode 100644
index 0000000..5730116
--- /dev/null
+++ b/server/src/api/config/notifications/mod.rs
@@ -0,0 +1,102 @@
+use anyhow::Error;
+use pdm_api_types::PRIV_SYS_AUDIT;
+use proxmox_router::{
+    list_subdirs_api_method, ApiMethod, Permission, Router, RpcEnvironment, SubdirMap,
+};
+use proxmox_schema::api;
+use proxmox_sortable_macro::sortable;
+use serde::Serialize;
+use serde_json::Value;
+
+pub mod gotify;
+pub mod matchers;
+pub mod sendmail;
+pub mod smtp;
+pub mod targets;
+pub mod webhook;
+
+#[sortable]
+const SUBDIRS: SubdirMap = &sorted!([
+    ("endpoints", &ENDPOINT_ROUTER),
+    ("matcher-fields", &FIELD_ROUTER),
+    ("matcher-field-values", &VALUE_ROUTER),
+    ("targets", &targets::ROUTER),
+    ("matchers", &matchers::ROUTER),
+]);
+
+pub const ROUTER: Router = Router::new()
+    .get(&list_subdirs_api_method!(SUBDIRS))
+    .subdirs(SUBDIRS);
+
+#[sortable]
+const ENDPOINT_SUBDIRS: SubdirMap = &sorted!([
+    ("gotify", &gotify::ROUTER),
+    ("sendmail", &sendmail::ROUTER),
+    ("smtp", &smtp::ROUTER),
+    ("webhook", &webhook::ROUTER),
+]);
+
+const ENDPOINT_ROUTER: Router = Router::new()
+    .get(&list_subdirs_api_method!(ENDPOINT_SUBDIRS))
+    .subdirs(ENDPOINT_SUBDIRS);
+
+const FIELD_ROUTER: Router = Router::new().get(&API_METHOD_GET_FIELDS);
+const VALUE_ROUTER: Router = Router::new().get(&API_METHOD_GET_VALUES);
+
+#[api]
+#[derive(Serialize)]
+/// A matchable field
+pub struct MatchableField {
+    /// Name of the field
+    name: String,
+}
+
+#[api]
+#[derive(Serialize)]
+/// A matchable metadata field value
+pub struct MatchableValue {
+    /// Field this value belongs to.
+    field: String,
+    /// Notification metadata value known by the system.
+    value: String,
+    /// Additional comment for this value.
+    comment: Option<String>,
+}
+
+#[api(
+    input: {
+        properties: {}
+    },
+    returns: {
+        description: "List of known metadata fields.",
+        type: Array,
+        items: {type: MatchableField},
+    },
+    access: {permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false)},
+)]
+/// Get all known metadata fields.
+pub fn get_fields() -> Result<Vec<MatchableField>, Error> {
+    Ok(vec![])
+}
+
+#[api(
+    input: {
+        properties: {},
+    },
+    returns: {
+        description: "List of known metadata field values.",
+        type: Array,
+        items: { type: MatchableValue },
+    },
+    access: {
+        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false),
+    },
+)]
+/// List all known, matchable metadata field values
+pub fn get_values(
+    _param: Value,
+    _info: &ApiMethod,
+    _rpcenv: &mut dyn RpcEnvironment,
+) -> Result<Vec<MatchableValue>, Error> {
+    Ok(vec![])
+}
diff --git a/server/src/api/config/notifications/sendmail.rs b/server/src/api/config/notifications/sendmail.rs
new file mode 100644
index 0000000..66c9656
--- /dev/null
+++ b/server/src/api/config/notifications/sendmail.rs
@@ -0,0 +1,173 @@
+use anyhow::Error;
+use serde_json::Value;
+
+use proxmox_notify::endpoints::sendmail::{
+    DeleteableSendmailProperty, SendmailConfig, SendmailConfigUpdater,
+};
+use proxmox_notify::schema::ENTITY_NAME_SCHEMA;
+use proxmox_router::{Permission, Router, RpcEnvironment};
+use proxmox_schema::api;
+
+use pdm_api_types::{PRIV_SYS_AUDIT, PRIV_SYS_MODIFY, PROXMOX_CONFIG_DIGEST_SCHEMA};
+
+#[api(
+    input: {
+        properties: {},
+    },
+    returns: {
+        description: "List of sendmail endpoints.",
+        type: Array,
+        items: { type: SendmailConfig },
+    },
+    access: {
+        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false),
+    },
+)]
+/// List all sendmail endpoints.
+pub fn list_endpoints(
+    _param: Value,
+    _rpcenv: &mut dyn RpcEnvironment,
+) -> Result<Vec<SendmailConfig>, Error> {
+    let config = pdm_config::notifications::config()?;
+
+    let endpoints = proxmox_notify::api::sendmail::get_endpoints(&config)?;
+
+    Ok(endpoints)
+}
+
+#[api(
+    input: {
+        properties: {
+            name: {
+                schema: ENTITY_NAME_SCHEMA,
+            }
+        },
+    },
+    returns: { type: SendmailConfig },
+    access: {
+        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false),
+    },
+)]
+/// Get a sendmail endpoint.
+pub fn get_endpoint(
+    name: String,
+    rpcenv: &mut dyn RpcEnvironment,
+) -> Result<SendmailConfig, Error> {
+    let config = pdm_config::notifications::config()?;
+    let endpoint = proxmox_notify::api::sendmail::get_endpoint(&config, &name)?;
+
+    rpcenv["digest"] = hex::encode(config.digest()).into();
+
+    Ok(endpoint)
+}
+
+#[api(
+    input: {
+        properties: {
+            endpoint: {
+                type: SendmailConfig,
+                flatten: true,
+            }
+        },
+    },
+    access: {
+        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
+    },
+)]
+/// Add a new sendmail endpoint.
+pub fn add_endpoint(
+    endpoint: SendmailConfig,
+    _rpcenv: &mut dyn RpcEnvironment,
+) -> Result<(), Error> {
+    let _lock = pdm_config::notifications::lock_config()?;
+    let mut config = pdm_config::notifications::config()?;
+
+    proxmox_notify::api::sendmail::add_endpoint(&mut config, endpoint)?;
+
+    pdm_config::notifications::save_config(config)?;
+    Ok(())
+}
+
+#[api(
+    input: {
+        properties: {
+            name: {
+                schema: ENTITY_NAME_SCHEMA,
+            },
+            updater: {
+                type: SendmailConfigUpdater,
+                flatten: true,
+            },
+            delete: {
+                description: "List of properties to delete.",
+                type: Array,
+                optional: true,
+                items: {
+                    type: DeleteableSendmailProperty,
+                }
+            },
+            digest: {
+                optional: true,
+                schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
+            },
+        },
+    },
+    access: {
+        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
+    },
+)]
+/// Update sendmail endpoint.
+pub fn update_endpoint(
+    name: String,
+    updater: SendmailConfigUpdater,
+    delete: Option<Vec<DeleteableSendmailProperty>>,
+    digest: Option<String>,
+    _rpcenv: &mut dyn RpcEnvironment,
+) -> Result<(), Error> {
+    let _lock = pdm_config::notifications::lock_config()?;
+    let mut config = pdm_config::notifications::config()?;
+    let digest = digest.map(hex::decode).transpose()?;
+
+    proxmox_notify::api::sendmail::update_endpoint(
+        &mut config,
+        &name,
+        updater,
+        delete.as_deref(),
+        digest.as_deref(),
+    )?;
+
+    pdm_config::notifications::save_config(config)?;
+    Ok(())
+}
+
+#[api(
+    input: {
+        properties: {
+            name: {
+                schema: ENTITY_NAME_SCHEMA,
+            }
+        },
+    },
+    access: {
+        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
+    },
+)]
+/// Delete sendmail endpoint.
+pub fn delete_endpoint(name: String, _rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
+    let _lock = pdm_config::notifications::lock_config()?;
+    let mut config = pdm_config::notifications::config()?;
+    proxmox_notify::api::sendmail::delete_endpoint(&mut config, &name)?;
+
+    pdm_config::notifications::save_config(config)?;
+    Ok(())
+}
+
+const ITEM_ROUTER: Router = Router::new()
+    .get(&API_METHOD_GET_ENDPOINT)
+    .put(&API_METHOD_UPDATE_ENDPOINT)
+    .delete(&API_METHOD_DELETE_ENDPOINT);
+
+pub const ROUTER: Router = Router::new()
+    .get(&API_METHOD_LIST_ENDPOINTS)
+    .post(&API_METHOD_ADD_ENDPOINT)
+    .match_all("name", &ITEM_ROUTER);
diff --git a/server/src/api/config/notifications/smtp.rs b/server/src/api/config/notifications/smtp.rs
new file mode 100644
index 0000000..4a48256
--- /dev/null
+++ b/server/src/api/config/notifications/smtp.rs
@@ -0,0 +1,186 @@
+use anyhow::Error;
+use serde_json::Value;
+
+use proxmox_notify::endpoints::smtp::{
+    DeleteableSmtpProperty, SmtpConfig, SmtpConfigUpdater, SmtpPrivateConfig,
+    SmtpPrivateConfigUpdater,
+};
+use proxmox_notify::schema::ENTITY_NAME_SCHEMA;
+use proxmox_router::{Permission, Router, RpcEnvironment};
+use proxmox_schema::api;
+
+use pdm_api_types::{PRIV_SYS_AUDIT, PRIV_SYS_MODIFY, PROXMOX_CONFIG_DIGEST_SCHEMA};
+
+#[api(
+    input: {
+        properties: {},
+    },
+    returns: {
+        description: "List of smtp endpoints.",
+        type: Array,
+        items: { type: SmtpConfig },
+    },
+    access: {
+        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false),
+    },
+)]
+/// List all smtp endpoints.
+pub fn list_endpoints(
+    _param: Value,
+    _rpcenv: &mut dyn RpcEnvironment,
+) -> Result<Vec<SmtpConfig>, Error> {
+    let config = pdm_config::notifications::config()?;
+
+    let endpoints = proxmox_notify::api::smtp::get_endpoints(&config)?;
+
+    Ok(endpoints)
+}
+
+#[api(
+    input: {
+        properties: {
+            name: {
+                schema: ENTITY_NAME_SCHEMA,
+            }
+        },
+    },
+    returns: { type: SmtpConfig },
+    access: {
+        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false),
+    },
+)]
+/// Get a smtp endpoint.
+pub fn get_endpoint(name: String, rpcenv: &mut dyn RpcEnvironment) -> Result<SmtpConfig, Error> {
+    let config = pdm_config::notifications::config()?;
+    let endpoint = proxmox_notify::api::smtp::get_endpoint(&config, &name)?;
+
+    rpcenv["digest"] = hex::encode(config.digest()).into();
+
+    Ok(endpoint)
+}
+
+#[api(
+    input: {
+        properties: {
+            endpoint: {
+                type: SmtpConfig,
+                flatten: true,
+            },
+            password: {
+                optional: true,
+                description: "SMTP authentication password"
+            }
+        },
+    },
+    access: {
+        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
+    },
+)]
+/// Add a new smtp endpoint.
+pub fn add_endpoint(
+    endpoint: SmtpConfig,
+    password: Option<String>,
+    _rpcenv: &mut dyn RpcEnvironment,
+) -> Result<(), Error> {
+    let _lock = pdm_config::notifications::lock_config()?;
+    let mut config = pdm_config::notifications::config()?;
+    let private_endpoint_config = SmtpPrivateConfig {
+        name: endpoint.name.clone(),
+        password,
+    };
+
+    proxmox_notify::api::smtp::add_endpoint(&mut config, endpoint, private_endpoint_config)?;
+
+    pdm_config::notifications::save_config(config)?;
+    Ok(())
+}
+
+#[api(
+    input: {
+        properties: {
+            name: {
+                schema: ENTITY_NAME_SCHEMA,
+            },
+            updater: {
+                type: SmtpConfigUpdater,
+                flatten: true,
+            },
+            password: {
+                description: "SMTP authentication password",
+                optional: true,
+            },
+            delete: {
+                description: "List of properties to delete.",
+                type: Array,
+                optional: true,
+                items: {
+                    type: DeleteableSmtpProperty,
+                }
+            },
+            digest: {
+                optional: true,
+                schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
+            },
+        },
+    },
+    access: {
+        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
+    },
+)]
+/// Update smtp endpoint.
+pub fn update_endpoint(
+    name: String,
+    updater: SmtpConfigUpdater,
+    password: Option<String>,
+    delete: Option<Vec<DeleteableSmtpProperty>>,
+    digest: Option<String>,
+    _rpcenv: &mut dyn RpcEnvironment,
+) -> Result<(), Error> {
+    let _lock = pdm_config::notifications::lock_config()?;
+    let mut config = pdm_config::notifications::config()?;
+    let digest = digest.map(hex::decode).transpose()?;
+
+    proxmox_notify::api::smtp::update_endpoint(
+        &mut config,
+        &name,
+        updater,
+        SmtpPrivateConfigUpdater { password },
+        delete.as_deref(),
+        digest.as_deref(),
+    )?;
+
+    pdm_config::notifications::save_config(config)?;
+    Ok(())
+}
+
+#[api(
+    input: {
+        properties: {
+            name: {
+                schema: ENTITY_NAME_SCHEMA,
+            }
+        },
+    },
+    access: {
+        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
+    },
+)]
+/// Delete smtp endpoint.
+pub fn delete_endpoint(name: String, _rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
+    let _lock = pdm_config::notifications::lock_config()?;
+    let mut config = pdm_config::notifications::config()?;
+    proxmox_notify::api::smtp::delete_endpoint(&mut config, &name)?;
+
+    pdm_config::notifications::save_config(config)?;
+    Ok(())
+}
+
+const ITEM_ROUTER: Router = Router::new()
+    .get(&API_METHOD_GET_ENDPOINT)
+    .put(&API_METHOD_UPDATE_ENDPOINT)
+    .delete(&API_METHOD_DELETE_ENDPOINT);
+
+pub const ROUTER: Router = Router::new()
+    .get(&API_METHOD_LIST_ENDPOINTS)
+    .post(&API_METHOD_ADD_ENDPOINT)
+    .match_all("name", &ITEM_ROUTER);
diff --git a/server/src/api/config/notifications/targets.rs b/server/src/api/config/notifications/targets.rs
new file mode 100644
index 0000000..b2655fb
--- /dev/null
+++ b/server/src/api/config/notifications/targets.rs
@@ -0,0 +1,61 @@
+use anyhow::Error;
+use serde_json::Value;
+
+use proxmox_notify::api::Target;
+use proxmox_notify::schema::ENTITY_NAME_SCHEMA;
+use proxmox_router::{list_subdirs_api_method, Permission, Router, RpcEnvironment, SubdirMap};
+use proxmox_schema::api;
+use proxmox_sortable_macro::sortable;
+
+use pdm_api_types::{PRIV_SYS_AUDIT, PRIV_SYS_MODIFY};
+
+#[api(
+    input: {
+        properties: {},
+    },
+    returns: {
+        description: "List of all entities which can be used as notification targets.",
+        type: Array,
+        items: { type: Target },
+    },
+    access: {
+        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false),
+    },
+)]
+/// List all notification targets.
+pub fn list_targets(_param: Value, _rpcenv: &mut dyn RpcEnvironment) -> Result<Vec<Target>, Error> {
+    let config = pdm_config::notifications::config()?;
+    let targets = proxmox_notify::api::get_targets(&config)?;
+
+    Ok(targets)
+}
+
+#[api(
+    input: {
+        properties: {
+            name: {
+                schema: ENTITY_NAME_SCHEMA,
+            },
+        }
+    },
+    access: {
+        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
+    },
+)]
+/// Test a given notification target.
+pub fn test_target(name: String, _rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
+    let config = pdm_config::notifications::config()?;
+    proxmox_notify::api::common::test_target(&config, &name)?;
+    Ok(())
+}
+
+#[sortable]
+const SUBDIRS: SubdirMap = &sorted!([("test", &TEST_ROUTER),]);
+const TEST_ROUTER: Router = Router::new().post(&API_METHOD_TEST_TARGET);
+const ITEM_ROUTER: Router = Router::new()
+    .get(&list_subdirs_api_method!(SUBDIRS))
+    .subdirs(SUBDIRS);
+
+pub const ROUTER: Router = Router::new()
+    .get(&API_METHOD_LIST_TARGETS)
+    .match_all("name", &ITEM_ROUTER);
diff --git a/server/src/api/config/notifications/webhook.rs b/server/src/api/config/notifications/webhook.rs
new file mode 100644
index 0000000..751086f
--- /dev/null
+++ b/server/src/api/config/notifications/webhook.rs
@@ -0,0 +1,170 @@
+use anyhow::Error;
+use serde_json::Value;
+
+use proxmox_notify::endpoints::webhook::{
+    DeleteableWebhookProperty, WebhookConfig, WebhookConfigUpdater,
+};
+use proxmox_notify::schema::ENTITY_NAME_SCHEMA;
+use proxmox_router::{Permission, Router, RpcEnvironment};
+use proxmox_schema::api;
+
+use pdm_api_types::{PRIV_SYS_AUDIT, PRIV_SYS_MODIFY, PROXMOX_CONFIG_DIGEST_SCHEMA};
+
+#[api(
+    input: {
+        properties: {},
+    },
+    returns: {
+        description: "List of webhook endpoints.",
+        type: Array,
+        items: { type: WebhookConfig },
+    },
+    access: {
+        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false),
+    },
+)]
+/// List all webhook endpoints.
+pub fn list_endpoints(
+    _param: Value,
+    _rpcenv: &mut dyn RpcEnvironment,
+) -> Result<Vec<WebhookConfig>, Error> {
+    let config = pdm_config::notifications::config()?;
+
+    let endpoints = proxmox_notify::api::webhook::get_endpoints(&config)?;
+
+    Ok(endpoints)
+}
+
+#[api(
+    input: {
+        properties: {
+            name: {
+                schema: ENTITY_NAME_SCHEMA,
+            }
+        },
+    },
+    returns: { type: WebhookConfig },
+    access: {
+        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false),
+    },
+)]
+/// Get a webhook endpoint.
+pub fn get_endpoint(name: String, rpcenv: &mut dyn RpcEnvironment) -> Result<WebhookConfig, Error> {
+    let config = pdm_config::notifications::config()?;
+    let endpoint = proxmox_notify::api::webhook::get_endpoint(&config, &name)?;
+
+    rpcenv["digest"] = hex::encode(config.digest()).into();
+
+    Ok(endpoint)
+}
+
+#[api(
+    input: {
+        properties: {
+            endpoint: {
+                type: WebhookConfig,
+                flatten: true,
+            },
+        },
+    },
+    access: {
+        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
+    },
+)]
+/// Add a new webhook endpoint.
+pub fn add_endpoint(
+    endpoint: WebhookConfig,
+    _rpcenv: &mut dyn RpcEnvironment,
+) -> Result<(), Error> {
+    let _lock = pdm_config::notifications::lock_config()?;
+    let mut config = pdm_config::notifications::config()?;
+
+    proxmox_notify::api::webhook::add_endpoint(&mut config, endpoint)?;
+
+    pdm_config::notifications::save_config(config)?;
+    Ok(())
+}
+
+#[api(
+    input: {
+        properties: {
+            name: {
+                schema: ENTITY_NAME_SCHEMA,
+            },
+            updater: {
+                type: WebhookConfigUpdater,
+                flatten: true,
+            },
+            delete: {
+                description: "List of properties to delete.",
+                type: Array,
+                optional: true,
+                items: {
+                    type: DeleteableWebhookProperty,
+                }
+            },
+            digest: {
+                optional: true,
+                schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
+            },
+        },
+    },
+    access: {
+        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
+    },
+)]
+/// Update webhook endpoint.
+pub fn update_endpoint(
+    name: String,
+    updater: WebhookConfigUpdater,
+    delete: Option<Vec<DeleteableWebhookProperty>>,
+    digest: Option<String>,
+    _rpcenv: &mut dyn RpcEnvironment,
+) -> Result<(), Error> {
+    let _lock = pdm_config::notifications::lock_config()?;
+    let mut config = pdm_config::notifications::config()?;
+    let digest = digest.map(hex::decode).transpose()?;
+
+    proxmox_notify::api::webhook::update_endpoint(
+        &mut config,
+        &name,
+        updater,
+        delete.as_deref(),
+        digest.as_deref(),
+    )?;
+
+    pdm_config::notifications::save_config(config)?;
+    Ok(())
+}
+
+#[api(
+    input: {
+        properties: {
+            name: {
+                schema: ENTITY_NAME_SCHEMA,
+            }
+        },
+    },
+    access: {
+        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
+    },
+)]
+/// Delete webhook endpoint.
+pub fn delete_endpoint(name: String, _rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
+    let _lock = pdm_config::notifications::lock_config()?;
+    let mut config = pdm_config::notifications::config()?;
+    proxmox_notify::api::webhook::delete_endpoint(&mut config, &name)?;
+
+    pdm_config::notifications::save_config(config)?;
+    Ok(())
+}
+
+const ITEM_ROUTER: Router = Router::new()
+    .get(&API_METHOD_GET_ENDPOINT)
+    .put(&API_METHOD_UPDATE_ENDPOINT)
+    .delete(&API_METHOD_DELETE_ENDPOINT);
+
+pub const ROUTER: Router = Router::new()
+    .get(&API_METHOD_LIST_ENDPOINTS)
+    .post(&API_METHOD_ADD_ENDPOINT)
+    .match_all("name", &ITEM_ROUTER);
diff --git a/server/src/context.rs b/server/src/context.rs
index c5da0af..acff3a8 100644
--- a/server/src/context.rs
+++ b/server/src/context.rs
@@ -38,5 +38,7 @@ pub fn init() -> Result<(), Error> {
         default_remote_setup();
     }
 
+    crate::notifications::init();
+
     Ok(())
 }
diff --git a/server/src/lib.rs b/server/src/lib.rs
index 5ed10d6..9723bc4 100644
--- a/server/src/lib.rs
+++ b/server/src/lib.rs
@@ -7,6 +7,7 @@ pub mod context;
 pub mod env;
 pub mod jobstate;
 pub mod metric_collection;
+pub mod notifications;
 pub mod parallel_fetcher;
 pub mod remote_cache;
 pub mod remote_tasks;
diff --git a/server/src/notifications/mod.rs b/server/src/notifications/mod.rs
new file mode 100644
index 0000000..e38584b
--- /dev/null
+++ b/server/src/notifications/mod.rs
@@ -0,0 +1,115 @@
+use std::path::Path;
+
+use proxmox_access_control::types::User;
+use proxmox_notify::context::Context;
+use proxmox_notify::renderer::TemplateSource;
+use proxmox_notify::Error;
+use tracing::error;
+
+use pdm_buildcfg::configdir;
+
+const PDM_NODE_CFG_FILENAME: &str = configdir!("/node.cfg");
+
+const DEFAULT_CONFIG: &str = "\
+sendmail: mail-to-root
+    comment Send mails to root@pam's email address
+    mailto-user root@pam
+
+
+matcher: default-matcher
+    mode all
+    target mail-to-root
+    comment Route all notifications to mail-to-root
+";
+
+fn attempt_file_read<P: AsRef<Path>>(path: P) -> Option<String> {
+    match proxmox_sys::fs::file_read_optional_string(path) {
+        Ok(contents) => contents,
+        Err(err) => {
+            error!("{err}");
+            None
+        }
+    }
+}
+
+fn lookup_datacenter_config_key(content: &str, key: &str) -> Option<String> {
+    let key_prefix = format!("{key}:");
+    let value = content
+        .lines()
+        .find_map(|line| line.strip_prefix(&key_prefix))?
+        .trim();
+
+    if value.is_empty() {
+        None
+    } else {
+        Some(value.to_string())
+    }
+}
+
+#[derive(Debug)]
+struct PDMContext;
+
+static PDM_CONTEXT: PDMContext = PDMContext;
+
+impl Context for PDMContext {
+    fn lookup_email_for_user(&self, user: &str) -> Option<String> {
+        let (config, _digest) = match proxmox_access_control::user::config() {
+            Ok(c) => c,
+            Err(err) => {
+                error!("failed to read user config: {err}");
+                return None;
+            }
+        };
+
+        match config.lookup::<User>("user", user) {
+            Ok(user) => user.email,
+            Err(_) => None,
+        }
+    }
+
+    fn default_sendmail_author(&self) -> String {
+        format!("Proxmox Datacenter Manager - {}", proxmox_sys::nodename())
+    }
+
+    fn default_sendmail_from(&self) -> String {
+        let content = attempt_file_read(PDM_NODE_CFG_FILENAME);
+        content
+            .and_then(|content| lookup_datacenter_config_key(&content, "email-from"))
+            .unwrap_or_else(|| String::from("root"))
+    }
+
+    fn http_proxy_config(&self) -> Option<String> {
+        let content = attempt_file_read(PDM_NODE_CFG_FILENAME);
+        content.and_then(|content| lookup_datacenter_config_key(&content, "http-proxy"))
+    }
+
+    fn default_config(&self) -> &'static str {
+        DEFAULT_CONFIG
+    }
+
+    fn lookup_template(
+        &self,
+        filename: &str,
+        namespace: Option<&str>,
+        source: TemplateSource,
+    ) -> Result<Option<String>, Error> {
+        let base = match source {
+            TemplateSource::Vendor => "/usr/share/proxmox-datacenter-manager/templates",
+            TemplateSource::Override => configdir!("/notification-templates"),
+        };
+
+        let path = Path::new(base)
+            .join(namespace.unwrap_or("default"))
+            .join(filename);
+
+        error!("{path:?}");
+
+        proxmox_sys::fs::file_read_optional_string(path)
+            .map_err(|err| Error::Generic(format!("could not load template: {err}")))
+    }
+}
+
+/// Initialize `proxmox-notify` by registering the PDM context.
+pub fn init() {
+    proxmox_notify::context::set_context(&PDM_CONTEXT);
+}
diff --git a/templates/Makefile b/templates/Makefile
new file mode 100644
index 0000000..294fa7b
--- /dev/null
+++ b/templates/Makefile
@@ -0,0 +1,14 @@
+include ../defines.mk
+
+NOTIFICATION_TEMPLATES=						\
+	default/test-body.txt.hbs				\
+	default/test-subject.txt.hbs			\
+
+all:
+
+clean:
+
+install:
+	install -dm755 $(DESTDIR)$(DATAROOTDIR)/proxmox-datacenter-manager/templates/default
+	$(foreach i,$(NOTIFICATION_TEMPLATES), \
+	    install -m644 $(i) $(DESTDIR)$(DATAROOTDIR)/proxmox-datacenter-manager/templates/$(i) ;)
diff --git a/templates/default/test-body.txt.hbs b/templates/default/test-body.txt.hbs
new file mode 100644
index 0000000..2445443
--- /dev/null
+++ b/templates/default/test-body.txt.hbs
@@ -0,0 +1 @@
+This is a test of the notification target '{{target}}'.
diff --git a/templates/default/test-subject.txt.hbs b/templates/default/test-subject.txt.hbs
new file mode 100644
index 0000000..cb8e132
--- /dev/null
+++ b/templates/default/test-subject.txt.hbs
@@ -0,0 +1 @@
+Test notification
-- 
2.47.3

Re: [PATCH datacenter-manager v2] Add notifications backend

[Shannon Sterz]

On Thu Apr 9, 2026 at 6:57 AM CEST, Arthur Bied-Charreton wrote:

nit: imo a commit message like "server: add notification backend" would
be more appropriate. usually it is recommended to try add a tag prefix
of what your code touches [1].

[1]: https://pve.proxmox.com/wiki/Developer_Documentation#Commits_and_Commit_Messages

> Ideally, the whole router should be in proxmox-notify. However, after
> discussing this quite extensively off-list with Lukas, we came to the
> conclusion that doing so would require a large refactor that is
> especially tricky given the fact that PVE would still need a different
> entry point into proxmox-notify than PBS and PDM, since it defines its
> router on the Perl side.

nit: usually mentions of offline discussions would be more appropriate
in the notes of a patch or a cover letter, than the commit message.
generally, it may make sense to split this up a little and add a cover
letter imo.

> For this reason, factoring the routing logic out into proxmox-notify is
> offset to a future where PVE also has a Rust-based router, and for now
> it is just copied over from PBS (commit: fbf8d1b2) with the adaptations
> described below.
>
> The `proxmox_notify::context::Context` trait implementation is added to
> PDM directly instead of creating a pdm.rs module in proxmox-notify to
> avoid adding even more product-specific logic to it. It uses
> `proxmox-access-control`, which is pulled in by PDM anyway, for getting
> the user config instead of reading the file directly.
>
> The `matcher-fields` and `matcher-field-values` endpoints currently
> return empty vecs, they will be updated in a future commit introducing
> actual notifications. The same applies for the notification templates.

any reason to not go ahead and add at least one notification in this
series too? being able to test that notifications function, but not
having anything to be notified about seems a little odd to me. one easy
notification you could add is the `send_certificate_renewal_mail` in
`api::nodes::certificates::spawn_certificate_worker()`. that should
essentially be identical to pbs anyway.

> The endpoints are made unprotected and the configuration files
> read/writable by `www-data`, like for `remotes.cfg`.
>
> Signed-off-by: Arthur Bied-Charreton <a.bied-charreton@proxmox.com>
> ---
>  Cargo.toml                                    |   1 +
>  Makefile                                      |   3 +-
>  debian/proxmox-datacenter-manager.install     |   2 +
>  defines.mk                                    |   1 +
>  lib/pdm-config/Cargo.toml                     |   1 +
>  lib/pdm-config/src/lib.rs                     |   1 +
>  lib/pdm-config/src/notifications.rs           |  39 ++++
>  server/Cargo.toml                             |   1 +
>  server/src/api/config/mod.rs                  |   2 +
>  server/src/api/config/notifications/gotify.rs | 185 +++++++++++++++++
>  .../src/api/config/notifications/matchers.rs  | 165 ++++++++++++++++
>  server/src/api/config/notifications/mod.rs    | 102 ++++++++++
>  .../src/api/config/notifications/sendmail.rs  | 173 ++++++++++++++++
>  server/src/api/config/notifications/smtp.rs   | 186 ++++++++++++++++++
>  .../src/api/config/notifications/targets.rs   |  61 ++++++
>  .../src/api/config/notifications/webhook.rs   | 170 ++++++++++++++++
>  server/src/context.rs                         |   2 +
>  server/src/lib.rs                             |   1 +
>  server/src/notifications/mod.rs               | 115 +++++++++++
>  templates/Makefile                            |  14 ++
>  templates/default/test-body.txt.hbs           |   1 +
>  templates/default/test-subject.txt.hbs        |   1 +
>  22 files changed, 1226 insertions(+), 1 deletion(-)
>  create mode 100644 lib/pdm-config/src/notifications.rs
>  create mode 100644 server/src/api/config/notifications/gotify.rs
>  create mode 100644 server/src/api/config/notifications/matchers.rs
>  create mode 100644 server/src/api/config/notifications/mod.rs
>  create mode 100644 server/src/api/config/notifications/sendmail.rs
>  create mode 100644 server/src/api/config/notifications/smtp.rs
>  create mode 100644 server/src/api/config/notifications/targets.rs
>  create mode 100644 server/src/api/config/notifications/webhook.rs
>  create mode 100644 server/src/notifications/mod.rs
>  create mode 100644 templates/Makefile
>  create mode 100644 templates/default/test-body.txt.hbs
>  create mode 100644 templates/default/test-subject.txt.hbs
>
> diff --git a/Cargo.toml b/Cargo.toml
> index ec2aa3d..918e831 100644
> --- a/Cargo.toml
> +++ b/Cargo.toml
> @@ -47,6 +47,7 @@ proxmox-ldap = { version = "1.1", features = ["sync"] }
>  proxmox-lang = "1.1"
>  proxmox-log = "1"
>  proxmox-login = "1.0.2"
> +proxmox-notify = "1"
>  proxmox-rest-server = "1"
>  # some use "cli", some use "cli" and "server", pbs-config uses nothing
>  proxmox-router = { version = "3.0.0", default-features = false }
> diff --git a/Makefile b/Makefile
> index 8e1cd13..48327c9 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -79,6 +79,7 @@ install: $(COMPILED_BINS) $(SHELL_COMPLETION_FILES)
>  	    install -m644 $(COMPLETION_DIR)/$(i) $(DESTDIR)$(ZSHCOMPDIR)/ ;)
>  	make -C services install
>  	$(MAKE) -C docs install
> +	$(MAKE) -C templates install
>
>  $(COMPILED_BINS) $(COMPILEDIR)/docgen &:
>  	$(CARGO) build $(CARGO_BUILD_ARGS)
> @@ -99,7 +100,7 @@ cargo-build:
>  $(BUILDDIR):
>  	rm -rf $@ $@.tmp
>  	mkdir $@.tmp
> -	cp -a debian/ server/ services/ cli/ lib/ docs/ ui/ defines.mk Makefile Cargo.toml $@.tmp
> +	cp -a debian/ server/ services/ cli/ lib/ docs/ ui/ templates/ defines.mk Makefile Cargo.toml $@.tmp
>  	echo "git clone git://git.proxmox.com/git/$(PACKAGE).git\\ngit checkout $$(git rev-parse HEAD)" \
>  	    > $@.tmp/debian/SOURCE
>  	mv $@.tmp $@
> diff --git a/debian/proxmox-datacenter-manager.install b/debian/proxmox-datacenter-manager.install
> index fad3f4a..fb2b1f1 100644
> --- a/debian/proxmox-datacenter-manager.install
> +++ b/debian/proxmox-datacenter-manager.install
> @@ -20,3 +20,5 @@ usr/share/man/man5/remotes.cfg.5
>  usr/share/man/man5/views.cfg.5
>  usr/share/zsh/vendor-completions/_pdmAtoB
>  usr/share/zsh/vendor-completions/_proxmox-datacenter-manager-admin
> +usr/share/proxmox-datacenter-manager/templates/default/test-body.txt.hbs
> +usr/share/proxmox-datacenter-manager/templates/default/test-subject.txt.hbs
> diff --git a/defines.mk b/defines.mk
> index 1b9c1a8..111923f 100644
> --- a/defines.mk
> +++ b/defines.mk
> @@ -3,6 +3,7 @@ BINDIR = $(PREFIX)/bin
>  SBINDIR = $(PREFIX)/sbin
>  LIBDIR = $(PREFIX)/lib
>  LIBEXECDIR = $(PREFIX)/libexec
> +DATAROOTDIR = $(PREFIX)/share
>  BASHCOMPDIR = $(PREFIX)/share/bash-completion/completions
>  ZSHCOMPDIR = $(PREFIX)/share/zsh/vendor-completions
>  MAN1DIR = $(PREFIX)/share/man/man1
> diff --git a/lib/pdm-config/Cargo.toml b/lib/pdm-config/Cargo.toml
> index d39c2ad..b6f3739 100644
> --- a/lib/pdm-config/Cargo.toml
> +++ b/lib/pdm-config/Cargo.toml
> @@ -23,5 +23,6 @@ proxmox-shared-memory.workspace = true
>  proxmox-simple-config.workspace = true
>  proxmox-sys = { workspace = true, features = [ "acl", "crypt", "timer" ] }
>  proxmox-acme-api.workspace = true
> +proxmox-notify.workspace = true
>  pdm-api-types.workspace = true
>  pdm-buildcfg.workspace = true
> diff --git a/lib/pdm-config/src/lib.rs b/lib/pdm-config/src/lib.rs
> index 4c49054..03dc247 100644
> --- a/lib/pdm-config/src/lib.rs
> +++ b/lib/pdm-config/src/lib.rs
> @@ -5,6 +5,7 @@ pub use pdm_buildcfg::{BACKUP_GROUP_NAME, BACKUP_USER_NAME};
>  pub mod certificate_config;
>  pub mod domains;
>  pub mod node;
> +pub mod notifications;
>  pub mod remotes;
>  pub mod setup;
>  pub mod views;
> diff --git a/lib/pdm-config/src/notifications.rs b/lib/pdm-config/src/notifications.rs
> new file mode 100644
> index 0000000..fd6ec79
> --- /dev/null
> +++ b/lib/pdm-config/src/notifications.rs
> @@ -0,0 +1,39 @@
> +use anyhow::Error;
> +
> +use proxmox_notify::Config;
> +
> +use pdm_buildcfg::configdir;
> +use proxmox_product_config::{open_api_lockfile, replace_config, ApiLockGuard};
> +use proxmox_sys::fs::file_read_optional_string;
> +
> +/// Configuration file location for notification targets/matchers.
> +pub const NOTIFICATION_CONFIG_PATH: &str = configdir!("/notifications.cfg");
> +
> +/// Private configuration file location for secrets - only readable by `root`.
> +pub const NOTIFICATION_PRIV_CONFIG_PATH: &str = configdir!("/notifications-priv.cfg");
> +
> +/// Lockfile to prevent concurrent write access.
> +pub const NOTIFICATION_LOCK_FILE: &str = configdir!("/.notifications.lck");
> +

you might want to move these to their own "notifications" sub-folder or
similar (maybe just "notify"?). in pdm we tend to put config files in
topically fitting sub-folders (e.g. access, acme, auth)

> +/// Get exclusive lock for `notifications.cfg`.
> +pub fn lock_config() -> Result<ApiLockGuard, Error> {
> +    open_api_lockfile(NOTIFICATION_LOCK_FILE, None, true)
> +}
> +
> +/// Load notification config.
> +pub fn config() -> Result<Config, Error> {
> +    let content = file_read_optional_string(NOTIFICATION_CONFIG_PATH)?.unwrap_or_default();
> +
> +    let priv_content =
> +        file_read_optional_string(NOTIFICATION_PRIV_CONFIG_PATH)?.unwrap_or_default();
> +
> +    Ok(Config::new(&content, &priv_content)?)

you might want to consider returning a `ConfigDigest` here too.
returning that encourages validating the digest as its less easily
forgotten.

> +}
> +
> +/// Save notification config.
> +pub fn save_config(config: Config) -> Result<(), Error> {
> +    let (cfg, priv_cfg) = config.write()?;
> +    replace_config(NOTIFICATION_CONFIG_PATH, cfg.as_bytes())?;
> +    replace_config(NOTIFICATION_PRIV_CONFIG_PATH, priv_cfg.as_bytes())?;

the privileged config should probably be saved with higher permission
than the general config. consider using `replace_secret_config` here.

> +    Ok(())
> +}
> diff --git a/server/Cargo.toml b/server/Cargo.toml
> index 6969549..a4f7bbd 100644
> --- a/server/Cargo.toml
> +++ b/server/Cargo.toml
> @@ -46,6 +46,7 @@ proxmox-lang.workspace = true
>  proxmox-ldap.workspace = true
>  proxmox-log.workspace = true
>  proxmox-login.workspace = true
> +proxmox-notify.workspace = true
>  proxmox-openid.workspace = true
>  proxmox-rest-server = { workspace = true, features = [ "templates" ] }
>  proxmox-router = { workspace = true, features = [ "cli", "server"] }
> diff --git a/server/src/api/config/mod.rs b/server/src/api/config/mod.rs
> index 8f646c1..a465219 100644
> --- a/server/src/api/config/mod.rs
> +++ b/server/src/api/config/mod.rs
> @@ -6,6 +6,7 @@ pub mod access;
>  pub mod acme;
>  pub mod certificate;
>  pub mod notes;
> +pub mod notifications;
>  pub mod views;
>
>  #[sortable]
> @@ -14,6 +15,7 @@ const SUBDIRS: SubdirMap = &sorted!([
>      ("acme", &acme::ROUTER),
>      ("certificate", &certificate::ROUTER),
>      ("notes", &notes::ROUTER),
> +    ("notifications", &notifications::ROUTER),
>      ("views", &views::ROUTER)
>  ]);
>
> diff --git a/server/src/api/config/notifications/gotify.rs b/server/src/api/config/notifications/gotify.rs
> new file mode 100644
> index 0000000..a281245
> --- /dev/null
> +++ b/server/src/api/config/notifications/gotify.rs
> @@ -0,0 +1,185 @@
> +use anyhow::Error;
> +use serde_json::Value;
> +
> +use proxmox_notify::endpoints::gotify::{
> +    DeleteableGotifyProperty, GotifyConfig, GotifyConfigUpdater, GotifyPrivateConfig,
> +    GotifyPrivateConfigUpdater,
> +};
> +use proxmox_notify::schema::ENTITY_NAME_SCHEMA;
> +use proxmox_router::{Permission, Router, RpcEnvironment};
> +use proxmox_schema::api;
> +
> +use pbs_api_types::{PRIV_SYS_AUDIT, PRIV_SYS_MODIFY, PROXMOX_CONFIG_DIGEST_SCHEMA};
> +
> +#[api(
> +    input: {
> +        properties: {},
> +    },
> +    returns: {
> +        description: "List of gotify endpoints.",
> +        type: Array,
> +        items: { type: GotifyConfig },
> +    },
> +    access: {
> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false),
> +    },
> +)]
> +/// List all gotify endpoints.
> +pub fn list_endpoints(
> +    _param: Value,
> +    _rpcenv: &mut dyn RpcEnvironment,
> +) -> Result<Vec<GotifyConfig>, Error> {
> +    let config = pdm_config::notifications::config()?;
> +
> +    let endpoints = proxmox_notify::api::gotify::get_endpoints(&config)?;
> +
> +    Ok(endpoints)
> +}
> +
> +#[api(
> +    input: {
> +        properties: {
> +            name: {
> +                schema: ENTITY_NAME_SCHEMA,
> +            }
> +        },
> +    },
> +    returns: { type: GotifyConfig },
> +    access: {
> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false),
> +    },
> +)]
> +/// Get a gotify endpoint.
> +pub fn get_endpoint(name: String, rpcenv: &mut dyn RpcEnvironment) -> Result<GotifyConfig, Error> {
> +    let config = pdm_config::notifications::config()?;
> +    let endpoint = proxmox_notify::api::gotify::get_endpoint(&config, &name)?;
> +
> +    rpcenv["digest"] = hex::encode(config.digest()).into();


if you return the digest as described above this could become:

    let (config, digest) = pdm_config::notifications::config()?;
    let endpoint = proxmox_notify::api::gotify::get_endpoint(&config, &name)?;

    rpcenv["digest"] = digest.to_hex().into();

> +
> +    Ok(endpoint)
> +}
> +
> +#[api(
> +    input: {
> +        properties: {
> +            endpoint: {
> +                type: GotifyConfig,
> +                flatten: true,
> +            },
> +            token: {
> +                description: "Authentication token",
> +            }
> +        },
> +    },
> +    access: {
> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
> +    },
> +)]
> +/// Add a new gotify endpoint.
> +pub fn add_endpoint(
> +    endpoint: GotifyConfig,
> +    token: String,
> +    _rpcenv: &mut dyn RpcEnvironment,
> +) -> Result<(), Error> {
> +    let _lock = pdm_config::notifications::lock_config()?;
> +    let mut config = pdm_config::notifications::config()?;

this should probably verify the digest. if you return a config
digest as described above this could become:

    let mut (config, expected_digest) = domains::config()?;
    expected_digest.detect_modification(digest.as_ref())?;

assuming that you take the digest as an input parameter to this api call
with:

    digest: {
        optional: true,
        type: ConfigDigest,
    }

and:

    digest: Option<ConfigDigest>

> +    let private_endpoint_config = GotifyPrivateConfig {
> +        name: endpoint.name.clone(),
> +        token,
> +    };
> +
> +    proxmox_notify::api::gotify::add_endpoint(&mut config, endpoint, private_endpoint_config)?;
> +
> +    pdm_config::notifications::save_config(config)?;
> +    Ok(())
> +}
> +
> +#[api(
> +    input: {
> +        properties: {
> +            name: {
> +                schema: ENTITY_NAME_SCHEMA,
> +            },
> +            updater: {
> +                type: GotifyConfigUpdater,
> +                flatten: true,
> +            },
> +            token: {
> +                description: "Authentication token",
> +                optional: true,
> +            },
> +            delete: {
> +                description: "List of properties to delete.",
> +                type: Array,
> +                optional: true,
> +                items: {
> +                    type: DeleteableGotifyProperty,
> +                }
> +            },
> +            digest: {
> +                optional: true,
> +                schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
> +            },
> +        },
> +    },
> +    access: {
> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
> +    },
> +)]
> +/// Update gotify endpoint.
> +pub fn update_endpoint(
> +    name: String,
> +    updater: GotifyConfigUpdater,
> +    token: Option<String>,
> +    delete: Option<Vec<DeleteableGotifyProperty>>,
> +    digest: Option<String>,
> +    _rpcenv: &mut dyn RpcEnvironment,
> +) -> Result<(), Error> {
> +    let _lock = pdm_config::notifications::lock_config()?;
> +    let mut config = pdm_config::notifications::config()?;
> +    let digest = digest.map(hex::decode).transpose()?;

using the `ConfigDigest` trait instead of `String` above you could drop
this line if im not mistaken.

> +
> +    proxmox_notify::api::gotify::update_endpoint(
> +        &mut config,
> +        &name,
> +        updater,
> +        GotifyPrivateConfigUpdater { token },
> +        delete.as_deref(),
> +        digest.as_deref(),
> +    )?;
> +
> +    pdm_config::notifications::save_config(config)?;
> +    Ok(())
> +}
> +
> +#[api(
> +    input: {
> +        properties: {
> +            name: {
> +                schema: ENTITY_NAME_SCHEMA,
> +            }
> +        },
> +    },
> +    access: {
> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
> +    },
> +)]
> +/// Delete gotify endpoint.
> +pub fn delete_endpoint(name: String, _rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
> +    let _lock = pdm_config::notifications::lock_config()?;
> +    let mut config = pdm_config::notifications::config()?;
> +    proxmox_notify::api::gotify::delete_gotify_endpoint(&mut config, &name)?;
> +

this would probably also benefit from checking the config digest.

> +    pdm_config::notifications::save_config(config)?;
> +    Ok(())
> +}
> +
> +const ITEM_ROUTER: Router = Router::new()
> +    .get(&API_METHOD_GET_ENDPOINT)
> +    .put(&API_METHOD_UPDATE_ENDPOINT)
> +    .delete(&API_METHOD_DELETE_ENDPOINT);
> +
> +pub const ROUTER: Router = Router::new()
> +    .get(&API_METHOD_LIST_ENDPOINTS)
> +    .post(&API_METHOD_ADD_ENDPOINT)
> +    .match_all("name", &ITEM_ROUTER);
> diff --git a/server/src/api/config/notifications/matchers.rs b/server/src/api/config/notifications/matchers.rs
> new file mode 100644
> index 0000000..9918305
> --- /dev/null
> +++ b/server/src/api/config/notifications/matchers.rs
> @@ -0,0 +1,165 @@
> +use anyhow::Error;
> +use serde_json::Value;
> +
> +use proxmox_notify::matcher::{DeleteableMatcherProperty, MatcherConfig, MatcherConfigUpdater};
> +use proxmox_notify::schema::ENTITY_NAME_SCHEMA;
> +use proxmox_router::{Permission, Router, RpcEnvironment};
> +use proxmox_schema::api;
> +
> +use pdm_api_types::{PRIV_SYS_AUDIT, PRIV_SYS_MODIFY, PROXMOX_CONFIG_DIGEST_SCHEMA};
> +
> +#[api(
> +    input: {
> +        properties: {},
> +    },
> +    returns: {
> +        description: "List of matchers.",
> +        type: Array,
> +        items: { type: MatcherConfig },
> +    },
> +    access: {
> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false),
> +    },
> +)]
> +/// List all notification matchers.
> +pub fn list_matchers(
> +    _param: Value,
> +    _rpcenv: &mut dyn RpcEnvironment,
> +) -> Result<Vec<MatcherConfig>, Error> {
> +    let config = pdm_config::notifications::config()?;
> +
> +    let matchers = proxmox_notify::api::matcher::get_matchers(&config)?;
> +
> +    Ok(matchers)
> +}
> +
> +#[api(
> +    input: {
> +        properties: {
> +            name: {
> +                schema: ENTITY_NAME_SCHEMA,
> +            }
> +        },
> +    },
> +    returns: { type: MatcherConfig },
> +    access: {
> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false),
> +    },
> +)]
> +/// Get a notification matcher.
> +pub fn get_matcher(name: String, rpcenv: &mut dyn RpcEnvironment) -> Result<MatcherConfig, Error> {
> +    let config = pdm_config::notifications::config()?;
> +    let matcher = proxmox_notify::api::matcher::get_matcher(&config, &name)?;
> +
> +    rpcenv["digest"] = hex::encode(config.digest()).into();

see above regarding `ConfigDigest`

> +
> +    Ok(matcher)
> +}
> +
> +#[api(
> +    input: {
> +        properties: {
> +            matcher: {
> +                type: MatcherConfig,
> +                flatten: true,
> +            }
> +        },
> +    },
> +    access: {
> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
> +    },
> +)]
> +/// Add a new notification matcher.
> +pub fn add_matcher(matcher: MatcherConfig, _rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
> +    let _lock = pdm_config::notifications::lock_config()?;
> +    let mut config = pdm_config::notifications::config()?;

see above regarding `ConfigDigest`

> +
> +    proxmox_notify::api::matcher::add_matcher(&mut config, matcher)?;
> +
> +    pdm_config::notifications::save_config(config)?;
> +    Ok(())
> +}
> +
> +#[api(
> +    input: {
> +        properties: {
> +            name: {
> +                schema: ENTITY_NAME_SCHEMA,
> +            },
> +            updater: {
> +                type: MatcherConfigUpdater,
> +                flatten: true,
> +            },
> +            delete: {
> +                description: "List of properties to delete.",
> +                type: Array,
> +                optional: true,
> +                items: {
> +                    type: DeleteableMatcherProperty,
> +                }
> +            },
> +            digest: {
> +                optional: true,
> +                schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
> +            },
> +        },
> +    },
> +    access: {
> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
> +    },
> +)]
> +/// Update notification matcher.
> +pub fn update_matcher(
> +    name: String,
> +    updater: MatcherConfigUpdater,
> +    delete: Option<Vec<DeleteableMatcherProperty>>,
> +    digest: Option<String>,
> +    _rpcenv: &mut dyn RpcEnvironment,
> +) -> Result<(), Error> {
> +    let _lock = pdm_config::notifications::lock_config()?;
> +    let mut config = pdm_config::notifications::config()?;
> +    let digest = digest.map(hex::decode).transpose()?;

see above regarding `ConfigDigest`.

> +
> +    proxmox_notify::api::matcher::update_matcher(
> +        &mut config,
> +        &name,
> +        updater,
> +        delete.as_deref(),
> +        digest.as_deref(),
> +    )?;
> +
> +    pdm_config::notifications::save_config(config)?;
> +    Ok(())
> +}
> +
> +#[api(
> +    input: {
> +        properties: {
> +            name: {
> +                schema: ENTITY_NAME_SCHEMA,
> +            }
> +        },
> +    },
> +    access: {
> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
> +    },
> +)]
> +/// Delete notification matcher.
> +pub fn delete_matcher(name: String, _rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
> +    let _lock = pdm_config::notifications::lock_config()?;
> +    let mut config = pdm_config::notifications::config()?;
> +    proxmox_notify::api::matcher::delete_matcher(&mut config, &name)?;
> +

see above regarding `ConfigDigest`.

> +    pdm_config::notifications::save_config(config)?;
> +    Ok(())
> +}
> +
> +const ITEM_ROUTER: Router = Router::new()
> +    .get(&API_METHOD_GET_MATCHER)
> +    .put(&API_METHOD_UPDATE_MATCHER)
> +    .delete(&API_METHOD_DELETE_MATCHER);
> +
> +pub const ROUTER: Router = Router::new()
> +    .get(&API_METHOD_LIST_MATCHERS)
> +    .post(&API_METHOD_ADD_MATCHER)
> +    .match_all("name", &ITEM_ROUTER);
> diff --git a/server/src/api/config/notifications/mod.rs b/server/src/api/config/notifications/mod.rs
> new file mode 100644
> index 0000000..5730116
> --- /dev/null
> +++ b/server/src/api/config/notifications/mod.rs
> @@ -0,0 +1,102 @@
> +use anyhow::Error;
> +use pdm_api_types::PRIV_SYS_AUDIT;
> +use proxmox_router::{
> +    list_subdirs_api_method, ApiMethod, Permission, Router, RpcEnvironment, SubdirMap,
> +};
> +use proxmox_schema::api;
> +use proxmox_sortable_macro::sortable;
> +use serde::Serialize;
> +use serde_json::Value;
> +
> +pub mod gotify;
> +pub mod matchers;
> +pub mod sendmail;
> +pub mod smtp;
> +pub mod targets;
> +pub mod webhook;
> +
> +#[sortable]
> +const SUBDIRS: SubdirMap = &sorted!([
> +    ("endpoints", &ENDPOINT_ROUTER),
> +    ("matcher-fields", &FIELD_ROUTER),
> +    ("matcher-field-values", &VALUE_ROUTER),
> +    ("targets", &targets::ROUTER),
> +    ("matchers", &matchers::ROUTER),
> +]);
> +
> +pub const ROUTER: Router = Router::new()
> +    .get(&list_subdirs_api_method!(SUBDIRS))
> +    .subdirs(SUBDIRS);
> +
> +#[sortable]
> +const ENDPOINT_SUBDIRS: SubdirMap = &sorted!([
> +    ("gotify", &gotify::ROUTER),
> +    ("sendmail", &sendmail::ROUTER),
> +    ("smtp", &smtp::ROUTER),
> +    ("webhook", &webhook::ROUTER),
> +]);
> +
> +const ENDPOINT_ROUTER: Router = Router::new()
> +    .get(&list_subdirs_api_method!(ENDPOINT_SUBDIRS))
> +    .subdirs(ENDPOINT_SUBDIRS);
> +
> +const FIELD_ROUTER: Router = Router::new().get(&API_METHOD_GET_FIELDS);
> +const VALUE_ROUTER: Router = Router::new().get(&API_METHOD_GET_VALUES);
> +
> +#[api]
> +#[derive(Serialize)]
> +/// A matchable field
> +pub struct MatchableField {
> +    /// Name of the field
> +    name: String,
> +}
> +
> +#[api]
> +#[derive(Serialize)]
> +/// A matchable metadata field value
> +pub struct MatchableValue {
> +    /// Field this value belongs to.
> +    field: String,
> +    /// Notification metadata value known by the system.
> +    value: String,
> +    /// Additional comment for this value.
> +    comment: Option<String>,
> +}
> +
> +#[api(
> +    input: {
> +        properties: {}
> +    },
> +    returns: {
> +        description: "List of known metadata fields.",
> +        type: Array,
> +        items: {type: MatchableField},
> +    },
> +    access: {permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false)},
> +)]
> +/// Get all known metadata fields.
> +pub fn get_fields() -> Result<Vec<MatchableField>, Error> {
> +    Ok(vec![])

if you decide to add the acme notification here in a next version, you
can add hostname here too imo.

> +}
> +
> +#[api(
> +    input: {
> +        properties: {},
> +    },
> +    returns: {
> +        description: "List of known metadata field values.",
> +        type: Array,
> +        items: { type: MatchableValue },
> +    },
> +    access: {
> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false),
> +    },
> +)]
> +/// List all known, matchable metadata field values
> +pub fn get_values(
> +    _param: Value,
> +    _info: &ApiMethod,
> +    _rpcenv: &mut dyn RpcEnvironment,
> +) -> Result<Vec<MatchableValue>, Error> {
> +    Ok(vec![])
> +}
> diff --git a/server/src/api/config/notifications/sendmail.rs b/server/src/api/config/notifications/sendmail.rs
> new file mode 100644
> index 0000000..66c9656
> --- /dev/null
> +++ b/server/src/api/config/notifications/sendmail.rs
> @@ -0,0 +1,173 @@
> +use anyhow::Error;
> +use serde_json::Value;
> +
> +use proxmox_notify::endpoints::sendmail::{
> +    DeleteableSendmailProperty, SendmailConfig, SendmailConfigUpdater,
> +};
> +use proxmox_notify::schema::ENTITY_NAME_SCHEMA;
> +use proxmox_router::{Permission, Router, RpcEnvironment};
> +use proxmox_schema::api;
> +
> +use pdm_api_types::{PRIV_SYS_AUDIT, PRIV_SYS_MODIFY, PROXMOX_CONFIG_DIGEST_SCHEMA};
> +
> +#[api(
> +    input: {
> +        properties: {},
> +    },
> +    returns: {
> +        description: "List of sendmail endpoints.",
> +        type: Array,
> +        items: { type: SendmailConfig },
> +    },
> +    access: {
> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false),
> +    },
> +)]
> +/// List all sendmail endpoints.
> +pub fn list_endpoints(
> +    _param: Value,
> +    _rpcenv: &mut dyn RpcEnvironment,
> +) -> Result<Vec<SendmailConfig>, Error> {
> +    let config = pdm_config::notifications::config()?;
> +
> +    let endpoints = proxmox_notify::api::sendmail::get_endpoints(&config)?;
> +
> +    Ok(endpoints)
> +}
> +
> +#[api(
> +    input: {
> +        properties: {
> +            name: {
> +                schema: ENTITY_NAME_SCHEMA,
> +            }
> +        },
> +    },
> +    returns: { type: SendmailConfig },
> +    access: {
> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false),
> +    },
> +)]
> +/// Get a sendmail endpoint.
> +pub fn get_endpoint(
> +    name: String,
> +    rpcenv: &mut dyn RpcEnvironment,
> +) -> Result<SendmailConfig, Error> {
> +    let config = pdm_config::notifications::config()?;
> +    let endpoint = proxmox_notify::api::sendmail::get_endpoint(&config, &name)?;
> +
> +    rpcenv["digest"] = hex::encode(config.digest()).into();

See above regarding `ConfigDigest`.

> +
> +    Ok(endpoint)
> +}
> +
> +#[api(
> +    input: {
> +        properties: {
> +            endpoint: {
> +                type: SendmailConfig,
> +                flatten: true,
> +            }
> +        },
> +    },
> +    access: {
> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
> +    },
> +)]
> +/// Add a new sendmail endpoint.
> +pub fn add_endpoint(
> +    endpoint: SendmailConfig,
> +    _rpcenv: &mut dyn RpcEnvironment,
> +) -> Result<(), Error> {
> +    let _lock = pdm_config::notifications::lock_config()?;
> +    let mut config = pdm_config::notifications::config()?;
> +

See above regarding `ConfigDigest`.

> +    proxmox_notify::api::sendmail::add_endpoint(&mut config, endpoint)?;
> +
> +    pdm_config::notifications::save_config(config)?;
> +    Ok(())
> +}
> +
> +#[api(
> +    input: {
> +        properties: {
> +            name: {
> +                schema: ENTITY_NAME_SCHEMA,
> +            },
> +            updater: {
> +                type: SendmailConfigUpdater,
> +                flatten: true,
> +            },
> +            delete: {
> +                description: "List of properties to delete.",
> +                type: Array,
> +                optional: true,
> +                items: {
> +                    type: DeleteableSendmailProperty,
> +                }
> +            },
> +            digest: {
> +                optional: true,
> +                schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
> +            },
> +        },
> +    },
> +    access: {
> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
> +    },
> +)]
> +/// Update sendmail endpoint.
> +pub fn update_endpoint(
> +    name: String,
> +    updater: SendmailConfigUpdater,
> +    delete: Option<Vec<DeleteableSendmailProperty>>,
> +    digest: Option<String>,
> +    _rpcenv: &mut dyn RpcEnvironment,
> +) -> Result<(), Error> {
> +    let _lock = pdm_config::notifications::lock_config()?;
> +    let mut config = pdm_config::notifications::config()?;
> +    let digest = digest.map(hex::decode).transpose()?;

See above regarding `ConfigDigest`.

> +
> +    proxmox_notify::api::sendmail::update_endpoint(
> +        &mut config,
> +        &name,
> +        updater,
> +        delete.as_deref(),
> +        digest.as_deref(),
> +    )?;
> +
> +    pdm_config::notifications::save_config(config)?;
> +    Ok(())
> +}
> +
> +#[api(
> +    input: {
> +        properties: {
> +            name: {
> +                schema: ENTITY_NAME_SCHEMA,
> +            }
> +        },
> +    },
> +    access: {
> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
> +    },
> +)]
> +/// Delete sendmail endpoint.
> +pub fn delete_endpoint(name: String, _rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
> +    let _lock = pdm_config::notifications::lock_config()?;
> +    let mut config = pdm_config::notifications::config()?;
> +    proxmox_notify::api::sendmail::delete_endpoint(&mut config, &name)?;
> +

See above regarding `ConfigDigest`.

> +    pdm_config::notifications::save_config(config)?;
> +    Ok(())
> +}
> +
> +const ITEM_ROUTER: Router = Router::new()
> +    .get(&API_METHOD_GET_ENDPOINT)
> +    .put(&API_METHOD_UPDATE_ENDPOINT)
> +    .delete(&API_METHOD_DELETE_ENDPOINT);
> +
> +pub const ROUTER: Router = Router::new()
> +    .get(&API_METHOD_LIST_ENDPOINTS)
> +    .post(&API_METHOD_ADD_ENDPOINT)
> +    .match_all("name", &ITEM_ROUTER);
> diff --git a/server/src/api/config/notifications/smtp.rs b/server/src/api/config/notifications/smtp.rs
> new file mode 100644
> index 0000000..4a48256
> --- /dev/null
> +++ b/server/src/api/config/notifications/smtp.rs
> @@ -0,0 +1,186 @@
> +use anyhow::Error;
> +use serde_json::Value;
> +
> +use proxmox_notify::endpoints::smtp::{
> +    DeleteableSmtpProperty, SmtpConfig, SmtpConfigUpdater, SmtpPrivateConfig,
> +    SmtpPrivateConfigUpdater,
> +};
> +use proxmox_notify::schema::ENTITY_NAME_SCHEMA;
> +use proxmox_router::{Permission, Router, RpcEnvironment};
> +use proxmox_schema::api;
> +
> +use pdm_api_types::{PRIV_SYS_AUDIT, PRIV_SYS_MODIFY, PROXMOX_CONFIG_DIGEST_SCHEMA};
> +
> +#[api(
> +    input: {
> +        properties: {},
> +    },
> +    returns: {
> +        description: "List of smtp endpoints.",
> +        type: Array,
> +        items: { type: SmtpConfig },
> +    },
> +    access: {
> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false),
> +    },
> +)]
> +/// List all smtp endpoints.
> +pub fn list_endpoints(
> +    _param: Value,
> +    _rpcenv: &mut dyn RpcEnvironment,
> +) -> Result<Vec<SmtpConfig>, Error> {
> +    let config = pdm_config::notifications::config()?;
> +
> +    let endpoints = proxmox_notify::api::smtp::get_endpoints(&config)?;
> +
> +    Ok(endpoints)
> +}
> +
> +#[api(
> +    input: {
> +        properties: {
> +            name: {
> +                schema: ENTITY_NAME_SCHEMA,
> +            }
> +        },
> +    },
> +    returns: { type: SmtpConfig },
> +    access: {
> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false),
> +    },
> +)]
> +/// Get a smtp endpoint.
> +pub fn get_endpoint(name: String, rpcenv: &mut dyn RpcEnvironment) -> Result<SmtpConfig, Error> {
> +    let config = pdm_config::notifications::config()?;
> +    let endpoint = proxmox_notify::api::smtp::get_endpoint(&config, &name)?;
> +
> +    rpcenv["digest"] = hex::encode(config.digest()).into();
> +

See above regarding `ConfigDigest`.

> +    Ok(endpoint)
> +}
> +
> +#[api(
> +    input: {
> +        properties: {
> +            endpoint: {
> +                type: SmtpConfig,
> +                flatten: true,
> +            },
> +            password: {
> +                optional: true,
> +                description: "SMTP authentication password"
> +            }
> +        },
> +    },
> +    access: {
> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
> +    },
> +)]
> +/// Add a new smtp endpoint.
> +pub fn add_endpoint(
> +    endpoint: SmtpConfig,
> +    password: Option<String>,
> +    _rpcenv: &mut dyn RpcEnvironment,
> +) -> Result<(), Error> {
> +    let _lock = pdm_config::notifications::lock_config()?;
> +    let mut config = pdm_config::notifications::config()?;
> +    let private_endpoint_config = SmtpPrivateConfig {
> +        name: endpoint.name.clone(),
> +        password,
> +    };
> +
> +    proxmox_notify::api::smtp::add_endpoint(&mut config, endpoint, private_endpoint_config)?;
> +

See above regarding `ConfigDigest`.

> +    pdm_config::notifications::save_config(config)?;
> +    Ok(())
> +}
> +
> +#[api(
> +    input: {
> +        properties: {
> +            name: {
> +                schema: ENTITY_NAME_SCHEMA,
> +            },
> +            updater: {
> +                type: SmtpConfigUpdater,
> +                flatten: true,
> +            },
> +            password: {
> +                description: "SMTP authentication password",
> +                optional: true,
> +            },
> +            delete: {
> +                description: "List of properties to delete.",
> +                type: Array,
> +                optional: true,
> +                items: {
> +                    type: DeleteableSmtpProperty,
> +                }
> +            },
> +            digest: {
> +                optional: true,
> +                schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
> +            },
> +        },
> +    },
> +    access: {
> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
> +    },
> +)]
> +/// Update smtp endpoint.
> +pub fn update_endpoint(
> +    name: String,
> +    updater: SmtpConfigUpdater,
> +    password: Option<String>,
> +    delete: Option<Vec<DeleteableSmtpProperty>>,
> +    digest: Option<String>,
> +    _rpcenv: &mut dyn RpcEnvironment,
> +) -> Result<(), Error> {
> +    let _lock = pdm_config::notifications::lock_config()?;
> +    let mut config = pdm_config::notifications::config()?;
> +    let digest = digest.map(hex::decode).transpose()?;

See above regarding `ConfigDigest`.

> +
> +    proxmox_notify::api::smtp::update_endpoint(
> +        &mut config,
> +        &name,
> +        updater,
> +        SmtpPrivateConfigUpdater { password },
> +        delete.as_deref(),
> +        digest.as_deref(),
> +    )?;
> +
> +    pdm_config::notifications::save_config(config)?;
> +    Ok(())
> +}
> +
> +#[api(
> +    input: {
> +        properties: {
> +            name: {
> +                schema: ENTITY_NAME_SCHEMA,
> +            }
> +        },
> +    },
> +    access: {
> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
> +    },
> +)]
> +/// Delete smtp endpoint.
> +pub fn delete_endpoint(name: String, _rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
> +    let _lock = pdm_config::notifications::lock_config()?;
> +    let mut config = pdm_config::notifications::config()?;
> +    proxmox_notify::api::smtp::delete_endpoint(&mut config, &name)?;

See above regarding `ConfigDigest`.

> +
> +    pdm_config::notifications::save_config(config)?;
> +    Ok(())
> +}
> +
> +const ITEM_ROUTER: Router = Router::new()
> +    .get(&API_METHOD_GET_ENDPOINT)
> +    .put(&API_METHOD_UPDATE_ENDPOINT)
> +    .delete(&API_METHOD_DELETE_ENDPOINT);
> +
> +pub const ROUTER: Router = Router::new()
> +    .get(&API_METHOD_LIST_ENDPOINTS)
> +    .post(&API_METHOD_ADD_ENDPOINT)
> +    .match_all("name", &ITEM_ROUTER);
> diff --git a/server/src/api/config/notifications/targets.rs b/server/src/api/config/notifications/targets.rs
> new file mode 100644
> index 0000000..b2655fb
> --- /dev/null
> +++ b/server/src/api/config/notifications/targets.rs
> @@ -0,0 +1,61 @@
> +use anyhow::Error;
> +use serde_json::Value;
> +
> +use proxmox_notify::api::Target;
> +use proxmox_notify::schema::ENTITY_NAME_SCHEMA;
> +use proxmox_router::{list_subdirs_api_method, Permission, Router, RpcEnvironment, SubdirMap};
> +use proxmox_schema::api;
> +use proxmox_sortable_macro::sortable;
> +
> +use pdm_api_types::{PRIV_SYS_AUDIT, PRIV_SYS_MODIFY};
> +
> +#[api(
> +    input: {
> +        properties: {},
> +    },
> +    returns: {
> +        description: "List of all entities which can be used as notification targets.",
> +        type: Array,
> +        items: { type: Target },
> +    },
> +    access: {
> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false),
> +    },
> +)]
> +/// List all notification targets.
> +pub fn list_targets(_param: Value, _rpcenv: &mut dyn RpcEnvironment) -> Result<Vec<Target>, Error> {
> +    let config = pdm_config::notifications::config()?;
> +    let targets = proxmox_notify::api::get_targets(&config)?;
> +
> +    Ok(targets)
> +}
> +
> +#[api(
> +    input: {
> +        properties: {
> +            name: {
> +                schema: ENTITY_NAME_SCHEMA,
> +            },
> +        }
> +    },
> +    access: {
> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
> +    },
> +)]
> +/// Test a given notification target.
> +pub fn test_target(name: String, _rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
> +    let config = pdm_config::notifications::config()?;
> +    proxmox_notify::api::common::test_target(&config, &name)?;
> +    Ok(())
> +}
> +
> +#[sortable]
> +const SUBDIRS: SubdirMap = &sorted!([("test", &TEST_ROUTER),]);
> +const TEST_ROUTER: Router = Router::new().post(&API_METHOD_TEST_TARGET);
> +const ITEM_ROUTER: Router = Router::new()
> +    .get(&list_subdirs_api_method!(SUBDIRS))
> +    .subdirs(SUBDIRS);
> +
> +pub const ROUTER: Router = Router::new()
> +    .get(&API_METHOD_LIST_TARGETS)
> +    .match_all("name", &ITEM_ROUTER);
> diff --git a/server/src/api/config/notifications/webhook.rs b/server/src/api/config/notifications/webhook.rs
> new file mode 100644
> index 0000000..751086f
> --- /dev/null
> +++ b/server/src/api/config/notifications/webhook.rs
> @@ -0,0 +1,170 @@
> +use anyhow::Error;
> +use serde_json::Value;
> +
> +use proxmox_notify::endpoints::webhook::{
> +    DeleteableWebhookProperty, WebhookConfig, WebhookConfigUpdater,
> +};
> +use proxmox_notify::schema::ENTITY_NAME_SCHEMA;
> +use proxmox_router::{Permission, Router, RpcEnvironment};
> +use proxmox_schema::api;
> +
> +use pdm_api_types::{PRIV_SYS_AUDIT, PRIV_SYS_MODIFY, PROXMOX_CONFIG_DIGEST_SCHEMA};
> +
> +#[api(
> +    input: {
> +        properties: {},
> +    },
> +    returns: {
> +        description: "List of webhook endpoints.",
> +        type: Array,
> +        items: { type: WebhookConfig },
> +    },
> +    access: {
> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false),
> +    },
> +)]
> +/// List all webhook endpoints.
> +pub fn list_endpoints(
> +    _param: Value,
> +    _rpcenv: &mut dyn RpcEnvironment,
> +) -> Result<Vec<WebhookConfig>, Error> {
> +    let config = pdm_config::notifications::config()?;
> +
> +    let endpoints = proxmox_notify::api::webhook::get_endpoints(&config)?;
> +
> +    Ok(endpoints)
> +}
> +
> +#[api(
> +    input: {
> +        properties: {
> +            name: {
> +                schema: ENTITY_NAME_SCHEMA,
> +            }
> +        },
> +    },
> +    returns: { type: WebhookConfig },
> +    access: {
> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false),
> +    },
> +)]
> +/// Get a webhook endpoint.
> +pub fn get_endpoint(name: String, rpcenv: &mut dyn RpcEnvironment) -> Result<WebhookConfig, Error> {
> +    let config = pdm_config::notifications::config()?;
> +    let endpoint = proxmox_notify::api::webhook::get_endpoint(&config, &name)?;
> +
> +    rpcenv["digest"] = hex::encode(config.digest()).into();

See above regarding `ConfigDigest`.

> +
> +    Ok(endpoint)
> +}
> +
> +#[api(
> +    input: {
> +        properties: {
> +            endpoint: {
> +                type: WebhookConfig,
> +                flatten: true,
> +            },
> +        },
> +    },
> +    access: {
> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
> +    },
> +)]
> +/// Add a new webhook endpoint.
> +pub fn add_endpoint(
> +    endpoint: WebhookConfig,
> +    _rpcenv: &mut dyn RpcEnvironment,
> +) -> Result<(), Error> {
> +    let _lock = pdm_config::notifications::lock_config()?;
> +    let mut config = pdm_config::notifications::config()?;

See above regarding `ConfigDigest`.

> +
> +    proxmox_notify::api::webhook::add_endpoint(&mut config, endpoint)?;
> +
> +    pdm_config::notifications::save_config(config)?;
> +    Ok(())
> +}
> +
> +#[api(
> +    input: {
> +        properties: {
> +            name: {
> +                schema: ENTITY_NAME_SCHEMA,
> +            },
> +            updater: {
> +                type: WebhookConfigUpdater,
> +                flatten: true,
> +            },
> +            delete: {
> +                description: "List of properties to delete.",
> +                type: Array,
> +                optional: true,
> +                items: {
> +                    type: DeleteableWebhookProperty,
> +                }
> +            },
> +            digest: {
> +                optional: true,
> +                schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
> +            },
> +        },
> +    },
> +    access: {
> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
> +    },
> +)]
> +/// Update webhook endpoint.
> +pub fn update_endpoint(
> +    name: String,
> +    updater: WebhookConfigUpdater,
> +    delete: Option<Vec<DeleteableWebhookProperty>>,
> +    digest: Option<String>,
> +    _rpcenv: &mut dyn RpcEnvironment,
> +) -> Result<(), Error> {
> +    let _lock = pdm_config::notifications::lock_config()?;
> +    let mut config = pdm_config::notifications::config()?;
> +    let digest = digest.map(hex::decode).transpose()?;

See above regarding `ConfigDigest`.

> +
> +    proxmox_notify::api::webhook::update_endpoint(
> +        &mut config,
> +        &name,
> +        updater,
> +        delete.as_deref(),
> +        digest.as_deref(),
> +    )?;
> +
> +    pdm_config::notifications::save_config(config)?;
> +    Ok(())
> +}
> +
> +#[api(
> +    input: {
> +        properties: {
> +            name: {
> +                schema: ENTITY_NAME_SCHEMA,
> +            }
> +        },
> +    },
> +    access: {
> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
> +    },
> +)]
> +/// Delete webhook endpoint.
> +pub fn delete_endpoint(name: String, _rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
> +    let _lock = pdm_config::notifications::lock_config()?;
> +    let mut config = pdm_config::notifications::config()?;
> +    proxmox_notify::api::webhook::delete_endpoint(&mut config, &name)?;

See above regarding `ConfigDigest`.

> +
> +    pdm_config::notifications::save_config(config)?;
> +    Ok(())
> +}
> +
> +const ITEM_ROUTER: Router = Router::new()
> +    .get(&API_METHOD_GET_ENDPOINT)
> +    .put(&API_METHOD_UPDATE_ENDPOINT)
> +    .delete(&API_METHOD_DELETE_ENDPOINT);
> +
> +pub const ROUTER: Router = Router::new()
> +    .get(&API_METHOD_LIST_ENDPOINTS)
> +    .post(&API_METHOD_ADD_ENDPOINT)
> +    .match_all("name", &ITEM_ROUTER);
> diff --git a/server/src/context.rs b/server/src/context.rs
> index c5da0af..acff3a8 100644
> --- a/server/src/context.rs
> +++ b/server/src/context.rs
> @@ -38,5 +38,7 @@ pub fn init() -> Result<(), Error> {
>          default_remote_setup();
>      }
>
> +    crate::notifications::init();
> +
>      Ok(())
>  }
> diff --git a/server/src/lib.rs b/server/src/lib.rs
> index 5ed10d6..9723bc4 100644
> --- a/server/src/lib.rs
> +++ b/server/src/lib.rs
> @@ -7,6 +7,7 @@ pub mod context;
>  pub mod env;
>  pub mod jobstate;
>  pub mod metric_collection;
> +pub mod notifications;
>  pub mod parallel_fetcher;
>  pub mod remote_cache;
>  pub mod remote_tasks;
> diff --git a/server/src/notifications/mod.rs b/server/src/notifications/mod.rs
> new file mode 100644
> index 0000000..e38584b
> --- /dev/null
> +++ b/server/src/notifications/mod.rs
> @@ -0,0 +1,115 @@
> +use std::path::Path;
> +
> +use proxmox_access_control::types::User;
> +use proxmox_notify::context::Context;
> +use proxmox_notify::renderer::TemplateSource;
> +use proxmox_notify::Error;
> +use tracing::error;
> +
> +use pdm_buildcfg::configdir;
> +
> +const PDM_NODE_CFG_FILENAME: &str = configdir!("/node.cfg");
> +
> +const DEFAULT_CONFIG: &str = "\
> +sendmail: mail-to-root
> +    comment Send mails to root@pam's email address
> +    mailto-user root@pam
> +
> +
> +matcher: default-matcher
> +    mode all
> +    target mail-to-root
> +    comment Route all notifications to mail-to-root
> +";
> +
> +fn attempt_file_read<P: AsRef<Path>>(path: P) -> Option<String> {
> +    match proxmox_sys::fs::file_read_optional_string(path) {
> +        Ok(contents) => contents,
> +        Err(err) => {
> +            error!("{err}");
> +            None
> +        }
> +    }
> +}
> +
> +fn lookup_datacenter_config_key(content: &str, key: &str) -> Option<String> {
> +    let key_prefix = format!("{key}:");
> +    let value = content
> +        .lines()
> +        .find_map(|line| line.strip_prefix(&key_prefix))?
> +        .trim();
> +
> +    if value.is_empty() {
> +        None
> +    } else {
> +        Some(value.to_string())
> +    }
> +}
> +
> +#[derive(Debug)]
> +struct PDMContext;
> +
> +static PDM_CONTEXT: PDMContext = PDMContext;
> +
> +impl Context for PDMContext {
> +    fn lookup_email_for_user(&self, user: &str) -> Option<String> {
> +        let (config, _digest) = match proxmox_access_control::user::config() {
> +            Ok(c) => c,
> +            Err(err) => {
> +                error!("failed to read user config: {err}");
> +                return None;
> +            }
> +        };
> +
> +        match config.lookup::<User>("user", user) {
> +            Ok(user) => user.email,
> +            Err(_) => None,
> +        }
> +    }
> +
> +    fn default_sendmail_author(&self) -> String {
> +        format!("Proxmox Datacenter Manager - {}", proxmox_sys::nodename())
> +    }
> +
> +    fn default_sendmail_from(&self) -> String {
> +        let content = attempt_file_read(PDM_NODE_CFG_FILENAME);
> +        content
> +            .and_then(|content| lookup_datacenter_config_key(&content, "email-from"))
> +            .unwrap_or_else(|| String::from("root"))
> +    }
> +
> +    fn http_proxy_config(&self) -> Option<String> {
> +        let content = attempt_file_read(PDM_NODE_CFG_FILENAME);
> +        content.and_then(|content| lookup_datacenter_config_key(&content, "http-proxy"))
> +    }
> +
> +    fn default_config(&self) -> &'static str {
> +        DEFAULT_CONFIG
> +    }
> +
> +    fn lookup_template(
> +        &self,
> +        filename: &str,
> +        namespace: Option<&str>,
> +        source: TemplateSource,
> +    ) -> Result<Option<String>, Error> {
> +        let base = match source {
> +            TemplateSource::Vendor => "/usr/share/proxmox-datacenter-manager/templates",
> +            TemplateSource::Override => configdir!("/notification-templates"),

if notification related config files are moved to their own sub-folder
this should probably become `notifications/templates` or similar

> +        };
> +
> +        let path = Path::new(base)
> +            .join(namespace.unwrap_or("default"))
> +            .join(filename);
> +
> +        error!("{path:?}");

is this intentional? i don't think unconditionally emitting an error
event here makes sense.

> +
> +        proxmox_sys::fs::file_read_optional_string(path)
> +            .map_err(|err| Error::Generic(format!("could not load template: {err}")))
> +    }
> +}
> +
> +/// Initialize `proxmox-notify` by registering the PDM context.
> +pub fn init() {
> +    proxmox_notify::context::set_context(&PDM_CONTEXT);
> +}
> diff --git a/templates/Makefile b/templates/Makefile
> new file mode 100644
> index 0000000..294fa7b
> --- /dev/null
> +++ b/templates/Makefile
> @@ -0,0 +1,14 @@
> +include ../defines.mk
> +
> +NOTIFICATION_TEMPLATES=						\
> +	default/test-body.txt.hbs				\
> +	default/test-subject.txt.hbs			\
> +
> +all:
> +
> +clean:
> +
> +install:
> +	install -dm755 $(DESTDIR)$(DATAROOTDIR)/proxmox-datacenter-manager/templates/default
> +	$(foreach i,$(NOTIFICATION_TEMPLATES), \
> +	    install -m644 $(i) $(DESTDIR)$(DATAROOTDIR)/proxmox-datacenter-manager/templates/$(i) ;)
> diff --git a/templates/default/test-body.txt.hbs b/templates/default/test-body.txt.hbs
> new file mode 100644
> index 0000000..2445443
> --- /dev/null
> +++ b/templates/default/test-body.txt.hbs
> @@ -0,0 +1 @@
> +This is a test of the notification target '{{target}}'.
> diff --git a/templates/default/test-subject.txt.hbs b/templates/default/test-subject.txt.hbs
> new file mode 100644
> index 0000000..cb8e132
> --- /dev/null
> +++ b/templates/default/test-subject.txt.hbs
> @@ -0,0 +1 @@
> +Test notification

Re: [PATCH datacenter-manager v2] Add notifications backend

[Arthur Bied-Charreton]

On Thu, Apr 09, 2026 at 11:20:22AM +0200, Shannon Sterz wrote:

Hey, thanks for the feedback! As a general answer, here is my reasoning 
as to why I did not make any improvemenets when porting the
notifications stack from PBS to PDM.

A possible goal Lukas and I discussed would be for proxmox-notify to 
include the router, to prevent the ~1000 lines of code duplication from 
spreading across 3 products.

This is currently not doable nicely because PVE defines its router on
the Perl side, meaning we would still need to have 2 different entry
points. Keeping this in mind though, I wanted to avoid the PDM router
implementation diverging from the PBS one too much in order to make an
eventual migration less painful.

I understand that this might be thinking too far into a future that will
possibly never happen, so I am of course happy to incorporate the 
changes you proposed.

> On Thu Apr 9, 2026 at 6:57 AM CEST, Arthur Bied-Charreton wrote:
> 
> nit: imo a commit message like "server: add notification backend" would
> be more appropriate. usually it is recommended to try add a tag prefix
> of what your code touches [1].
> 
> [1]: https://pve.proxmox.com/wiki/Developer_Documentation#Commits_and_Commit_Messages
> 
> > Ideally, the whole router should be in proxmox-notify. However, after
> > discussing this quite extensively off-list with Lukas, we came to the
> > conclusion that doing so would require a large refactor that is
> > especially tricky given the fact that PVE would still need a different
> > entry point into proxmox-notify than PBS and PDM, since it defines its
> > router on the Perl side.
> 
> nit: usually mentions of offline discussions would be more appropriate
> in the notes of a patch or a cover letter, than the commit message.
Noted, thanks!
> generally, it may make sense to split this up a little and add a cover
> letter imo.
> 
Yes, since this is currently 1:1 copy-pasted from PBS I figured it might
make sense to have it all in the same commit, but if we iterate on this,
I will of course split it up.

> > For this reason, factoring the routing logic out into proxmox-notify is
> > offset to a future where PVE also has a Rust-based router, and for now
> > it is just copied over from PBS (commit: fbf8d1b2) with the adaptations
> > described below.
> >
> > The `proxmox_notify::context::Context` trait implementation is added to
> > PDM directly instead of creating a pdm.rs module in proxmox-notify to
> > avoid adding even more product-specific logic to it. It uses
> > `proxmox-access-control`, which is pulled in by PDM anyway, for getting
> > the user config instead of reading the file directly.
> >
> > The `matcher-fields` and `matcher-field-values` endpoints currently
> > return empty vecs, they will be updated in a future commit introducing
> > actual notifications. The same applies for the notification templates.
> 
> any reason to not go ahead and add at least one notification in this
> series too? being able to test that notifications function, but not
> having anything to be notified about seems a little odd to me. one easy
> notification you could add is the `send_certificate_renewal_mail` in
> `api::nodes::certificates::spawn_certificate_worker()`. that should
> essentially be identical to pbs anyway.
> 
Yes that would have been better, I just wanted to get this out of the
way while I worked on adding notifications. Will add that in v2. 

[...]
> > +use anyhow::Error;
> > +
> > +use proxmox_notify::Config;
> > +
> > +use pdm_buildcfg::configdir;
> > +use proxmox_product_config::{open_api_lockfile, replace_config, ApiLockGuard};
> > +use proxmox_sys::fs::file_read_optional_string;
> > +
> > +/// Configuration file location for notification targets/matchers.
> > +pub const NOTIFICATION_CONFIG_PATH: &str = configdir!("/notifications.cfg");
> > +
> > +/// Private configuration file location for secrets - only readable by `root`.
> > +pub const NOTIFICATION_PRIV_CONFIG_PATH: &str = configdir!("/notifications-priv.cfg");
> > +
> > +/// Lockfile to prevent concurrent write access.
> > +pub const NOTIFICATION_LOCK_FILE: &str = configdir!("/.notifications.lck");
> > +
> 
> you might want to move these to their own "notifications" sub-folder or
> similar (maybe just "notify"?). in pdm we tend to put config files in
> topically fitting sub-folders (e.g. access, acme, auth)
> 
ACK, notify sounds good to me

> > +/// Get exclusive lock for `notifications.cfg`.
> > +pub fn lock_config() -> Result<ApiLockGuard, Error> {
> > +    open_api_lockfile(NOTIFICATION_LOCK_FILE, None, true)
> > +}
> > +
> > +/// Load notification config.
> > +pub fn config() -> Result<Config, Error> {
> > +    let content = file_read_optional_string(NOTIFICATION_CONFIG_PATH)?.unwrap_or_default();
> > +
> > +    let priv_content =
> > +        file_read_optional_string(NOTIFICATION_PRIV_CONFIG_PATH)?.unwrap_or_default();
> > +
> > +    Ok(Config::new(&content, &priv_content)?)
> 
> you might want to consider returning a `ConfigDigest` here too.
> returning that encourages validating the digest as its less easily
> forgotten.
> 
As written above, I made no improvements while porting this over from
PBS. I agree this would be better though. If we are okay diverging from
PBS I will be happy to add this to v2. (Note that I did not answer all
ConfigDigest-related comments on the handlers directly, please consider 
those all acknowledged, if we go with this I will of course address all of 
them).

> > +}
> > +
> > +/// Save notification config.
> > +pub fn save_config(config: Config) -> Result<(), Error> {
> > +    let (cfg, priv_cfg) = config.write()?;
> > +    replace_config(NOTIFICATION_CONFIG_PATH, cfg.as_bytes())?;
> > +    replace_config(NOTIFICATION_PRIV_CONFIG_PATH, priv_cfg.as_bytes())?;
> 
> the privileged config should probably be saved with higher permission
> than the general config. consider using `replace_secret_config` here.
> 
Yes that's what I initially had, but the remotes.cfg (which also stores
sensitive data like the remote access tokens) is also stored with 0640
and www-data:www-data, so I figured it would make sense to do the same
here, what do you think?

[...]
> > +/// Get a gotify endpoint.
> > +pub fn get_endpoint(name: String, rpcenv: &mut dyn RpcEnvironment) -> Result<GotifyConfig, Error> {
> > +    let config = pdm_config::notifications::config()?;
> > +    let endpoint = proxmox_notify::api::gotify::get_endpoint(&config, &name)?;
> > +
> > +    rpcenv["digest"] = hex::encode(config.digest()).into();
> 
> 
> if you return the digest as described above this could become:
> 
>     let (config, digest) = pdm_config::notifications::config()?;
>     let endpoint = proxmox_notify::api::gotify::get_endpoint(&config, &name)?;
> 
>     rpcenv["digest"] = digest.to_hex().into();
> 
ACK, see general answer on ConfigDigest above

[...]
> > +/// Add a new gotify endpoint.
> > +pub fn add_endpoint(
> > +    endpoint: GotifyConfig,
> > +    token: String,
> > +    _rpcenv: &mut dyn RpcEnvironment,
> > +) -> Result<(), Error> {
> > +    let _lock = pdm_config::notifications::lock_config()?;
> > +    let mut config = pdm_config::notifications::config()?;
> 
> this should probably verify the digest. if you return a config
> digest as described above this could become:
> 
>     let mut (config, expected_digest) = domains::config()?;
>     expected_digest.detect_modification(digest.as_ref())?;
> 
> assuming that you take the digest as an input parameter to this api call
> with:
> 
>     digest: {
>         optional: true,
>         type: ConfigDigest,
>     }
> 
> and:
> 
>     digest: Option<ConfigDigest>
> 
ACK, see general answer on ConfigDigest above

[...]
> > +#[api(
> > +    input: {
> > +        properties: {}
> > +    },
> > +    returns: {
> > +        description: "List of known metadata fields.",
> > +        type: Array,
> > +        items: {type: MatchableField},
> > +    },
> > +    access: {permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false)},
> > +)]
> > +/// Get all known metadata fields.
> > +pub fn get_fields() -> Result<Vec<MatchableField>, Error> {
> > +    Ok(vec![])
> 
> if you decide to add the acme notification here in a next version, you
> can add hostname here too imo.
>
ACK, will do
[...]
> > +        };
> > +
> > +        let path = Path::new(base)
> > +            .join(namespace.unwrap_or("default"))
> > +            .join(filename);
> > +
> > +        error!("{path:?}");
> 
> is this intentional? i don't think unconditionally emitting an error
> event here makes sense.
> 
No, I used that for debugging and forgot to remove it, sorry about
that...
> > +
> > +        proxmox_sys::fs::file_read_optional_string(path)
> > +            .map_err(|err| Error::Generic(format!("could not load template: {err}")))
> > +    }
> > +}
> > +
> > +/// Initialize `proxmox-notify` by registering the PDM context.
> > +pub fn init() {
> > +    proxmox_notify::context::set_context(&PDM_CONTEXT);
> > +}
> > diff --git a/templates/Makefile b/templates/Makefile
> > new file mode 100644
> > index 0000000..294fa7b
> > --- /dev/null
> > +++ b/templates/Makefile
> > @@ -0,0 +1,14 @@
> > +include ../defines.mk
> > +
> > +NOTIFICATION_TEMPLATES=						\
> > +	default/test-body.txt.hbs				\
> > +	default/test-subject.txt.hbs			\
> > +
> > +all:
> > +
> > +clean:
> > +
> > +install:
> > +	install -dm755 $(DESTDIR)$(DATAROOTDIR)/proxmox-datacenter-manager/templates/default
> > +	$(foreach i,$(NOTIFICATION_TEMPLATES), \
> > +	    install -m644 $(i) $(DESTDIR)$(DATAROOTDIR)/proxmox-datacenter-manager/templates/$(i) ;)
> > diff --git a/templates/default/test-body.txt.hbs b/templates/default/test-body.txt.hbs
> > new file mode 100644
> > index 0000000..2445443
> > --- /dev/null
> > +++ b/templates/default/test-body.txt.hbs
> > @@ -0,0 +1 @@
> > +This is a test of the notification target '{{target}}'.
> > diff --git a/templates/default/test-subject.txt.hbs b/templates/default/test-subject.txt.hbs
> > new file mode 100644
> > index 0000000..cb8e132
> > --- /dev/null
> > +++ b/templates/default/test-subject.txt.hbs
> > @@ -0,0 +1 @@
> > +Test notification
> 

Re: [PATCH datacenter-manager v2] Add notifications backend

[Shannon Sterz]

On Thu Apr 9, 2026 at 12:18 PM CEST, Arthur Bied-Charreton wrote:
> On Thu, Apr 09, 2026 at 11:20:22AM +0200, Shannon Sterz wrote:
>
> Hey, thanks for the feedback! As a general answer, here is my reasoning
> as to why I did not make any improvemenets when porting the
> notifications stack from PBS to PDM.
>
> A possible goal Lukas and I discussed would be for proxmox-notify to
> include the router, to prevent the ~1000 lines of code duplication from
> spreading across 3 products.
>
> This is currently not doable nicely because PVE defines its router on
> the Perl side, meaning we would still need to have 2 different entry
> points. Keeping this in mind though, I wanted to avoid the PDM router
> implementation diverging from the PBS one too much in order to make an
> eventual migration less painful.
>
> I understand that this might be thinking too far into a future that will
> possibly never happen, so I am of course happy to incorporate the
> changes you proposed.

i think updating the code to how we'd write it these days is still
reasonable, though. we'll have to eventually do that anyway when
factoring it out. as long as nothing changes on a functional level, this
shouldn't be to big of an issue.

-->8 snip 8<--

>
>> you might want to consider returning a `ConfigDigest` here too.
>> returning that encourages validating the digest as its less easily
>> forgotten.
>>
> As written above, I made no improvements while porting this over from
> PBS. I agree this would be better though. If we are okay diverging from
> PBS I will be happy to add this to v2. (Note that I did not answer all
> ConfigDigest-related comments on the handlers directly, please consider
> those all acknowledged, if we go with this I will of course address all of
> them).

that's fair they were pretty repetitive anyway. imo this is something we
should encourage earlier rather than later. updating pbs to support this
shouldn't be too much work once we do factor this out either imo.

>
>> > +}
>> > +
>> > +/// Save notification config.
>> > +pub fn save_config(config: Config) -> Result<(), Error> {
>> > +    let (cfg, priv_cfg) = config.write()?;
>> > +    replace_config(NOTIFICATION_CONFIG_PATH, cfg.as_bytes())?;
>> > +    replace_config(NOTIFICATION_PRIV_CONFIG_PATH, priv_cfg.as_bytes())?;
>>
>> the privileged config should probably be saved with higher permission
>> than the general config. consider using `replace_secret_config` here.
>>
> Yes that's what I initially had, but the remotes.cfg (which also stores
> sensitive data like the remote access tokens) is also stored with 0640
> and www-data:www-data, so I figured it would make sense to do the same
> here, what do you think?

unless necessary i'd default to stricter permissions from the outset.
that would follow the principle of least privilege.

the remotes.cfg is currently set to more lenient permissions because we
lack a mechanism to contact remotes in a more privileged context. this
is not the case for notifications, so keeping the stricter permissions
is preferred.

hope that makes sense :)

> [...]
>> > +/// Get a gotify endpoint.
>> > +pub fn get_endpoint(name: String, rpcenv: &mut dyn RpcEnvironment) -> Result<GotifyConfig, Error> {
>> > +    let config = pdm_config::notifications::config()?;
>> > +    let endpoint = proxmox_notify::api::gotify::get_endpoint(&config, &name)?;
>> > +
>> > +    rpcenv["digest"] = hex::encode(config.digest()).into();

-->8 snip 8<--

Re: [PATCH datacenter-manager v2] Add notifications backend

[Lukas Wagner]

On Thu Apr 9, 2026 at 11:20 AM CEST, Shannon Sterz wrote:
> On Thu Apr 9, 2026 at 6:57 AM CEST, Arthur Bied-Charreton wrote:
>
> nit: imo a commit message like "server: add notification backend" would
> be more appropriate. usually it is recommended to try add a tag prefix
> of what your code touches [1].
>
> [1]: https://pve.proxmox.com/wiki/Developer_Documentation#Commits_and_Commit_Messages
>

+1

>> Ideally, the whole router should be in proxmox-notify. However, after
>> discussing this quite extensively off-list with Lukas, we came to the
>> conclusion that doing so would require a large refactor that is
>> especially tricky given the fact that PVE would still need a different
>> entry point into proxmox-notify than PBS and PDM, since it defines its
>> router on the Perl side.
>
> nit: usually mentions of offline discussions would be more appropriate
> in the notes of a patch or a cover letter, than the commit message.
> generally, it may make sense to split this up a little and add a cover
> letter imo.
>

There is truly no single 'correct' way to split up such a change, but
this would probably be how I'd split it up:

 - pdm-config part as a separate commit
 - commit for defining/setting up notification context
 - commit for the 'test' notification template (since that one is needed
   by the 'test' route)
 - a commit for moving in the routes from PBS, as close to the original
   as possible (just to make it compile, e.g. replacing references to
   pbs_config with pdm_config, etc.)
 - if there are any improvements that are added on top of the PBS code,
   add them as commits on top (so that it is easier to backport them to
   PBS, if appropriate)

>> For this reason, factoring the routing logic out into proxmox-notify is
>> offset to a future where PVE also has a Rust-based router, and for now
>> it is just copied over from PBS (commit: fbf8d1b2) with the adaptations
>> described below.
>>
>> The `proxmox_notify::context::Context` trait implementation is added to
>> PDM directly instead of creating a pdm.rs module in proxmox-notify to
>> avoid adding even more product-specific logic to it. It uses
>> `proxmox-access-control`, which is pulled in by PDM anyway, for getting
>> the user config instead of reading the file directly.
>>
>> The `matcher-fields` and `matcher-field-values` endpoints currently
>> return empty vecs, they will be updated in a future commit introducing
>> actual notifications. The same applies for the notification templates.
>
> any reason to not go ahead and add at least one notification in this
> series too? being able to test that notifications function, but not
> having anything to be notified about seems a little odd to me. one easy
> notification you could add is the `send_certificate_renewal_mail` in
> `api::nodes::certificates::spawn_certificate_worker()`. that should
> essentially be identical to pbs anyway.
>

This was coordinated with me, I told him that it would be okay to send
this part early. He can either build on top of this series and include
more patches (e.g. adding the GUI, notification events, docs, etc.) in
future versions, or if it makes sense we can also apply certain parts
early.


>> The endpoints are made unprotected and the configuration files
>> read/writable by `www-data`, like for `remotes.cfg`.
>>
>> Signed-off-by: Arthur Bied-Charreton <a.bied-charreton@proxmox.com>
>> ---
>>  Cargo.toml                                    |   1 +
>>  Makefile                                      |   3 +-
>>  debian/proxmox-datacenter-manager.install     |   2 +
>>  defines.mk                                    |   1 +
>>  lib/pdm-config/Cargo.toml                     |   1 +
>>  lib/pdm-config/src/lib.rs                     |   1 +
>>  lib/pdm-config/src/notifications.rs           |  39 ++++
>>  server/Cargo.toml                             |   1 +
>>  server/src/api/config/mod.rs                  |   2 +
>>  server/src/api/config/notifications/gotify.rs | 185 +++++++++++++++++
>>  .../src/api/config/notifications/matchers.rs  | 165 ++++++++++++++++
>>  server/src/api/config/notifications/mod.rs    | 102 ++++++++++
>>  .../src/api/config/notifications/sendmail.rs  | 173 ++++++++++++++++
>>  server/src/api/config/notifications/smtp.rs   | 186 ++++++++++++++++++
>>  .../src/api/config/notifications/targets.rs   |  61 ++++++
>>  .../src/api/config/notifications/webhook.rs   | 170 ++++++++++++++++
>>  server/src/context.rs                         |   2 +
>>  server/src/lib.rs                             |   1 +
>>  server/src/notifications/mod.rs               | 115 +++++++++++
>>  templates/Makefile                            |  14 ++
>>  templates/default/test-body.txt.hbs           |   1 +
>>  templates/default/test-subject.txt.hbs        |   1 +
>>  22 files changed, 1226 insertions(+), 1 deletion(-)
>>  create mode 100644 lib/pdm-config/src/notifications.rs
>>  create mode 100644 server/src/api/config/notifications/gotify.rs
>>  create mode 100644 server/src/api/config/notifications/matchers.rs
>>  create mode 100644 server/src/api/config/notifications/mod.rs
>>  create mode 100644 server/src/api/config/notifications/sendmail.rs
>>  create mode 100644 server/src/api/config/notifications/smtp.rs
>>  create mode 100644 server/src/api/config/notifications/targets.rs
>>  create mode 100644 server/src/api/config/notifications/webhook.rs
>>  create mode 100644 server/src/notifications/mod.rs
>>  create mode 100644 templates/Makefile
>>  create mode 100644 templates/default/test-body.txt.hbs
>>  create mode 100644 templates/default/test-subject.txt.hbs
>>

[...]

>> +
>> +use pdm_buildcfg::configdir;
>> +use proxmox_product_config::{open_api_lockfile, replace_config, ApiLockGuard};
>> +use proxmox_sys::fs::file_read_optional_string;
>> +
>> +/// Configuration file location for notification targets/matchers.
>> +pub const NOTIFICATION_CONFIG_PATH: &str = configdir!("/notifications.cfg");
>> +
>> +/// Private configuration file location for secrets - only readable by `root`.
>> +pub const NOTIFICATION_PRIV_CONFIG_PATH: &str = configdir!("/notifications-priv.cfg");
>> +
>> +/// Lockfile to prevent concurrent write access.
>> +pub const NOTIFICATION_LOCK_FILE: &str = configdir!("/.notifications.lck");
>> +
>
> you might want to move these to their own "notifications" sub-folder or
> similar (maybe just "notify"?). in pdm we tend to put config files in
> topically fitting sub-folders (e.g. access, acme, auth)
>

Would be okay for me, but I'd prefer `notifications` for consistency
with what we do on other products.

>> +/// Get exclusive lock for `notifications.cfg`.
>> +pub fn lock_config() -> Result<ApiLockGuard, Error> {
>> +    open_api_lockfile(NOTIFICATION_LOCK_FILE, None, true)
>> +}
>> +
>> +/// Load notification config.
>> +pub fn config() -> Result<Config, Error> {
>> +    let content = file_read_optional_string(NOTIFICATION_CONFIG_PATH)?.unwrap_or_default();
>> +
>> +    let priv_content =
>> +        file_read_optional_string(NOTIFICATION_PRIV_CONFIG_PATH)?.unwrap_or_default();
>> +
>> +    Ok(Config::new(&content, &priv_content)?)
>
> you might want to consider returning a `ConfigDigest` here too.
> returning that encourages validating the digest as its less easily
> forgotten.

digest-checking is handled in the API handler implementations in
proxmox_notify::api where it seemed useful (see my later comments), it
cannot really be forgotten since it is a required parameter for these
functions.

I guess we could do a 

	let config = Config::new(&content, &priv_content)?;

	Ok((config, config.digest()))

here, but I'm not sure it gains us much?

>
>> +}
>> +
>> +/// Save notification config.
>> +pub fn save_config(config: Config) -> Result<(), Error> {
>> +    let (cfg, priv_cfg) = config.write()?;
>> +    replace_config(NOTIFICATION_CONFIG_PATH, cfg.as_bytes())?;
>> +    replace_config(NOTIFICATION_PRIV_CONFIG_PATH, priv_cfg.as_bytes())?;
>
> the privileged config should probably be saved with higher permission
> than the general config. consider using `replace_secret_config` here.
>

Unlike PBS, the notification system will run in the unprivileged process
in PDM, so that webhook/smtp stuff will not run as root. This means that
we also need to store the secrets with less strict permissions. We
already do the same for remotes.cfg and remotes.shadow, so this should
be fine here as well, I think.
Of course, we could also use the same approach as in PBS (notification
code runs as root, secrets are 0600 root:root), but I prefer the
approach used here.

>> +    Ok(())
>> +}
>> diff --git a/server/Cargo.toml b/server/Cargo.toml
>> index 6969549..a4f7bbd 100644
>> --- a/server/Cargo.toml
>> +++ b/server/Cargo.toml
>> @@ -46,6 +46,7 @@ proxmox-lang.workspace = true
>>  proxmox-ldap.workspace = true
>>  proxmox-log.workspace = true
>>  proxmox-login.workspace = true
>> +proxmox-notify.workspace = true
>>  proxmox-openid.workspace = true
>>  proxmox-rest-server = { workspace = true, features = [ "templates" ] }
>>  proxmox-router = { workspace = true, features = [ "cli", "server"] }
>> diff --git a/server/src/api/config/mod.rs b/server/src/api/config/mod.rs
>> index 8f646c1..a465219 100644
>> --- a/server/src/api/config/mod.rs
>> +++ b/server/src/api/config/mod.rs
>> @@ -6,6 +6,7 @@ pub mod access;
>>  pub mod acme;
>>  pub mod certificate;
>>  pub mod notes;
>> +pub mod notifications;
>>  pub mod views;
>>
>>  #[sortable]
>> @@ -14,6 +15,7 @@ const SUBDIRS: SubdirMap = &sorted!([
>>      ("acme", &acme::ROUTER),
>>      ("certificate", &certificate::ROUTER),
>>      ("notes", &notes::ROUTER),
>> +    ("notifications", &notifications::ROUTER),
>>      ("views", &views::ROUTER)
>>  ]);
>>
>> diff --git a/server/src/api/config/notifications/gotify.rs b/server/src/api/config/notifications/gotify.rs
>> new file mode 100644
>> index 0000000..a281245
>> --- /dev/null
>> +++ b/server/src/api/config/notifications/gotify.rs
>> @@ -0,0 +1,185 @@
>> +use anyhow::Error;
>> +use serde_json::Value;
>> +
>> +use proxmox_notify::endpoints::gotify::{
>> +    DeleteableGotifyProperty, GotifyConfig, GotifyConfigUpdater, GotifyPrivateConfig,
>> +    GotifyPrivateConfigUpdater,
>> +};
>> +use proxmox_notify::schema::ENTITY_NAME_SCHEMA;
>> +use proxmox_router::{Permission, Router, RpcEnvironment};
>> +use proxmox_schema::api;
>> +
>> +use pbs_api_types::{PRIV_SYS_AUDIT, PRIV_SYS_MODIFY, PROXMOX_CONFIG_DIGEST_SCHEMA};
>> +
>> +#[api(
>> +    input: {
>> +        properties: {},
>> +    },
>> +    returns: {
>> +        description: "List of gotify endpoints.",
>> +        type: Array,
>> +        items: { type: GotifyConfig },
>> +    },
>> +    access: {
>> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false),
>> +    },
>> +)]
>> +/// List all gotify endpoints.
>> +pub fn list_endpoints(
>> +    _param: Value,
>> +    _rpcenv: &mut dyn RpcEnvironment,
>> +) -> Result<Vec<GotifyConfig>, Error> {
>> +    let config = pdm_config::notifications::config()?;
>> +
>> +    let endpoints = proxmox_notify::api::gotify::get_endpoints(&config)?;
>> +
>> +    Ok(endpoints)
>> +}
>> +
>> +#[api(
>> +    input: {
>> +        properties: {
>> +            name: {
>> +                schema: ENTITY_NAME_SCHEMA,
>> +            }
>> +        },
>> +    },
>> +    returns: { type: GotifyConfig },
>> +    access: {
>> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false),
>> +    },
>> +)]
>> +/// Get a gotify endpoint.
>> +pub fn get_endpoint(name: String, rpcenv: &mut dyn RpcEnvironment) -> Result<GotifyConfig, Error> {
>> +    let config = pdm_config::notifications::config()?;
>> +    let endpoint = proxmox_notify::api::gotify::get_endpoint(&config, &name)?;
>> +
>> +    rpcenv["digest"] = hex::encode(config.digest()).into();
>
>
> if you return the digest as described above this could become:
>
>     let (config, digest) = pdm_config::notifications::config()?;
>     let endpoint = proxmox_notify::api::gotify::get_endpoint(&config, &name)?;
>
>     rpcenv["digest"] = digest.to_hex().into();
>

>> +
>> +    Ok(endpoint)
>> +}
>> +
>> +#[api(
>> +    input: {
>> +        properties: {
>> +            endpoint: {
>> +                type: GotifyConfig,
>> +                flatten: true,
>> +            },
>> +            token: {
>> +                description: "Authentication token",
>> +            }
>> +        },
>> +    },
>> +    access: {
>> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
>> +    },
>> +)]
>> +/// Add a new gotify endpoint.
>> +pub fn add_endpoint(
>> +    endpoint: GotifyConfig,
>> +    token: String,
>> +    _rpcenv: &mut dyn RpcEnvironment,
>> +) -> Result<(), Error> {
>> +    let _lock = pdm_config::notifications::lock_config()?;
>> +    let mut config = pdm_config::notifications::config()?;
>
> this should probably verify the digest. if you return a config
> digest as described above this could become:
>
>     let mut (config, expected_digest) = domains::config()?;
>     expected_digest.detect_modification(digest.as_ref())?;
>
> assuming that you take the digest as an input parameter to this api call
> with:
>
>     digest: {
>         optional: true,
>         type: ConfigDigest,
>     }
>
> and:
>
>     digest: Option<ConfigDigest>
>

We lock the config anyway, so it does not really matter (I think?) if
somebody else modified some other entity? And if somebody had added an
entity with the same id, then we fail anyways... So I'm not sure if
using the digest here gains us anything?

I quickly checked a couple add_* handlers in PBS, we don't
really check the digest when adding new entities there as well.

Generally, with regards to using the ConfigDigest type: +1
I think this type was introduced well after the notification API was
added to PBS, which might be the reason I did not use them there.

>> +    let private_endpoint_config = GotifyPrivateConfig {
>> +        name: endpoint.name.clone(),
>> +        token,
>> +    };
>> +
>> +    proxmox_notify::api::gotify::add_endpoint(&mut config, endpoint, private_endpoint_config)?;
>> +
>> +    pdm_config::notifications::save_config(config)?;
>> +    Ok(())
>> +}
>> +
>> +#[api(
>> +    input: {
>> +        properties: {
>> +            name: {
>> +                schema: ENTITY_NAME_SCHEMA,
>> +            },
>> +            updater: {
>> +                type: GotifyConfigUpdater,
>> +                flatten: true,
>> +            },
>> +            token: {
>> +                description: "Authentication token",
>> +                optional: true,
>> +            },
>> +            delete: {
>> +                description: "List of properties to delete.",
>> +                type: Array,
>> +                optional: true,
>> +                items: {
>> +                    type: DeleteableGotifyProperty,
>> +                }
>> +            },
>> +            digest: {
>> +                optional: true,
>> +                schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
>> +            },
>> +        },
>> +    },
>> +    access: {
>> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
>> +    },
>> +)]
>> +/// Update gotify endpoint.
>> +pub fn update_endpoint(
>> +    name: String,
>> +    updater: GotifyConfigUpdater,
>> +    token: Option<String>,
>> +    delete: Option<Vec<DeleteableGotifyProperty>>,
>> +    digest: Option<String>,
>> +    _rpcenv: &mut dyn RpcEnvironment,
>> +) -> Result<(), Error> {
>> +    let _lock = pdm_config::notifications::lock_config()?;
>> +    let mut config = pdm_config::notifications::config()?;
>> +    let digest = digest.map(hex::decode).transpose()?;
>
> using the `ConfigDigest` trait instead of `String` above you could drop
> this line if im not mistaken.
>
>> +
>> +    proxmox_notify::api::gotify::update_endpoint(
>> +        &mut config,
>> +        &name,
>> +        updater,
>> +        GotifyPrivateConfigUpdater { token },
>> +        delete.as_deref(),
>> +        digest.as_deref(),
>> +    )?;
>> +
>> +    pdm_config::notifications::save_config(config)?;
>> +    Ok(())
>> +}
>> +
>> +#[api(
>> +    input: {
>> +        properties: {
>> +            name: {
>> +                schema: ENTITY_NAME_SCHEMA,
>> +            }
>> +        },
>> +    },
>> +    access: {
>> +        permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false),
>> +    },
>> +)]
>> +/// Delete gotify endpoint.
>> +pub fn delete_endpoint(name: String, _rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
>> +    let _lock = pdm_config::notifications::lock_config()?;
>> +    let mut config = pdm_config::notifications::config()?;
>> +    proxmox_notify::api::gotify::delete_gotify_endpoint(&mut config, &name)?;
>> +
>
> this would probably also benefit from checking the config digest.
>

Same here, I'm not really sure what the benefit would be here?

IMO there are a couple cases to consider for concurrent modifications:
  - somebody modified entity A, we delete B -> should be fine
  - somebody modified entity A, we delete A -> does not matter, we want
    to delete it anyways
  - we deleted A, somebody else modifies A -> update fails anyways due
    to the wrong config digest or the entity missing already
  - both try to delete A -> fails for one of both due to the already
    deleted entity (HTTP 404)

Did I miss something?

[...]


>> +
>> +#[derive(Debug)]
>> +struct PDMContext;
>> +
>> +static PDM_CONTEXT: PDMContext = PDMContext;
>> +
>> +impl Context for PDMContext {
>> +    fn lookup_email_for_user(&self, user: &str) -> Option<String> {
>> +        let (config, _digest) = match proxmox_access_control::user::config() {
>> +            Ok(c) => c,
>> +            Err(err) => {
>> +                error!("failed to read user config: {err}");
>> +                return None;
>> +            }
>> +        };
>> +
>> +        match config.lookup::<User>("user", user) {
>> +            Ok(user) => user.email,
>> +            Err(_) => None,
>> +        }
>> +    }
>> +
>> +    fn default_sendmail_author(&self) -> String {
>> +        format!("Proxmox Datacenter Manager - {}", proxmox_sys::nodename())
>> +    }
>> +
>> +    fn default_sendmail_from(&self) -> String {
>> +        let content = attempt_file_read(PDM_NODE_CFG_FILENAME);
>> +        content
>> +            .and_then(|content| lookup_datacenter_config_key(&content, "email-from"))
>> +            .unwrap_or_else(|| String::from("root"))
>> +    }
>> +
>> +    fn http_proxy_config(&self) -> Option<String> {
>> +        let content = attempt_file_read(PDM_NODE_CFG_FILENAME);
>> +        content.and_then(|content| lookup_datacenter_config_key(&content, "http-proxy"))
>> +    }
>> +
>> +    fn default_config(&self) -> &'static str {
>> +        DEFAULT_CONFIG
>> +    }
>> +
>> +    fn lookup_template(
>> +        &self,
>> +        filename: &str,
>> +        namespace: Option<&str>,
>> +        source: TemplateSource,
>> +    ) -> Result<Option<String>, Error> {
>> +        let base = match source {
>> +            TemplateSource::Vendor => "/usr/share/proxmox-datacenter-manager/templates",

I've always regretted calling it just 'templates' and considered sending
a patch to rectify it for the other products. How about:

   /usr/share/proxmox-datacenter-manager/notifications/templates

Then it's also somewhat consistent with the proposal below.


>> +            TemplateSource::Override => configdir!("/notification-templates"),
>
> if notification related config files are moved to their own sub-folder
> this should probably become `notifications/templates` or similar
>

+1

Re: [PATCH datacenter-manager v2] Add notifications backend

[Shannon Sterz]

On Thu Apr 9, 2026 at 2:07 PM CEST, Lukas Wagner wrote:
> On Thu Apr 9, 2026 at 11:20 AM CEST, Shannon Sterz wrote:
>> On Thu Apr 9, 2026 at 6:57 AM CEST, Arthur Bied-Charreton wrote:

-->8 snip 8<--

>>
>> any reason to not go ahead and add at least one notification in this
>> series too? being able to test that notifications function, but not
>> having anything to be notified about seems a little odd to me. one easy
>> notification you could add is the `send_certificate_renewal_mail` in
>> `api::nodes::certificates::spawn_certificate_worker()`. that should
>> essentially be identical to pbs anyway.
>>
>
> This was coordinated with me, I told him that it would be okay to send
> this part early. He can either build on top of this series and include
> more patches (e.g. adding the GUI, notification events, docs, etc.) in
> future versions, or if it makes sense we can also apply certain parts
> early.
>

ack, was not aware of that.

>
>>> The endpoints are made unprotected and the configuration files
>>> read/writable by `www-data`, like for `remotes.cfg`.
>>>
>>> Signed-off-by: Arthur Bied-Charreton <a.bied-charreton@proxmox.com>
>>> ---
>>>  Cargo.toml                                    |   1 +
>>>  Makefile                                      |   3 +-
>>>  debian/proxmox-datacenter-manager.install     |   2 +
>>>  defines.mk                                    |   1 +
>>>  lib/pdm-config/Cargo.toml                     |   1 +
>>>  lib/pdm-config/src/lib.rs                     |   1 +
>>>  lib/pdm-config/src/notifications.rs           |  39 ++++
>>>  server/Cargo.toml                             |   1 +
>>>  server/src/api/config/mod.rs                  |   2 +
>>>  server/src/api/config/notifications/gotify.rs | 185 +++++++++++++++++
>>>  .../src/api/config/notifications/matchers.rs  | 165 ++++++++++++++++
>>>  server/src/api/config/notifications/mod.rs    | 102 ++++++++++
>>>  .../src/api/config/notifications/sendmail.rs  | 173 ++++++++++++++++
>>>  server/src/api/config/notifications/smtp.rs   | 186 ++++++++++++++++++
>>>  .../src/api/config/notifications/targets.rs   |  61 ++++++
>>>  .../src/api/config/notifications/webhook.rs   | 170 ++++++++++++++++
>>>  server/src/context.rs                         |   2 +
>>>  server/src/lib.rs                             |   1 +
>>>  server/src/notifications/mod.rs               | 115 +++++++++++
>>>  templates/Makefile                            |  14 ++
>>>  templates/default/test-body.txt.hbs           |   1 +
>>>  templates/default/test-subject.txt.hbs        |   1 +
>>>  22 files changed, 1226 insertions(+), 1 deletion(-)
>>>  create mode 100644 lib/pdm-config/src/notifications.rs
>>>  create mode 100644 server/src/api/config/notifications/gotify.rs
>>>  create mode 100644 server/src/api/config/notifications/matchers.rs
>>>  create mode 100644 server/src/api/config/notifications/mod.rs
>>>  create mode 100644 server/src/api/config/notifications/sendmail.rs
>>>  create mode 100644 server/src/api/config/notifications/smtp.rs
>>>  create mode 100644 server/src/api/config/notifications/targets.rs
>>>  create mode 100644 server/src/api/config/notifications/webhook.rs
>>>  create mode 100644 server/src/notifications/mod.rs
>>>  create mode 100644 templates/Makefile
>>>  create mode 100644 templates/default/test-body.txt.hbs
>>>  create mode 100644 templates/default/test-subject.txt.hbs
>>>
>
> [...]
>
>>> +
>>> +use pdm_buildcfg::configdir;
>>> +use proxmox_product_config::{open_api_lockfile, replace_config, ApiLockGuard};
>>> +use proxmox_sys::fs::file_read_optional_string;
>>> +
>>> +/// Configuration file location for notification targets/matchers.
>>> +pub const NOTIFICATION_CONFIG_PATH: &str = configdir!("/notifications.cfg");
>>> +
>>> +/// Private configuration file location for secrets - only readable by `root`.
>>> +pub const NOTIFICATION_PRIV_CONFIG_PATH: &str = configdir!("/notifications-priv.cfg");
>>> +
>>> +/// Lockfile to prevent concurrent write access.
>>> +pub const NOTIFICATION_LOCK_FILE: &str = configdir!("/.notifications.lck");
>>> +
>>
>> you might want to move these to their own "notifications" sub-folder or
>> similar (maybe just "notify"?). in pdm we tend to put config files in
>> topically fitting sub-folders (e.g. access, acme, auth)
>>
>
> Would be okay for me, but I'd prefer `notifications` for consistency
> with what we do on other products.
>

sounds fine to me, i mostly suggested "notify" because it's shorter.

>>> +/// Get exclusive lock for `notifications.cfg`.
>>> +pub fn lock_config() -> Result<ApiLockGuard, Error> {
>>> +    open_api_lockfile(NOTIFICATION_LOCK_FILE, None, true)
>>> +}
>>> +
>>> +/// Load notification config.
>>> +pub fn config() -> Result<Config, Error> {
>>> +    let content = file_read_optional_string(NOTIFICATION_CONFIG_PATH)?.unwrap_or_default();
>>> +
>>> +    let priv_content =
>>> +        file_read_optional_string(NOTIFICATION_PRIV_CONFIG_PATH)?.unwrap_or_default();
>>> +
>>> +    Ok(Config::new(&content, &priv_content)?)
>>
>> you might want to consider returning a `ConfigDigest` here too.
>> returning that encourages validating the digest as its less easily
>> forgotten.
>
> digest-checking is handled in the API handler implementations in
> proxmox_notify::api where it seemed useful (see my later comments), it
> cannot really be forgotten since it is a required parameter for these
> functions.

yes that's true for this crate, but from what i can tell returning a
config alongside its digest is a common pattern. but you are righ, in
this case we enforce it through `proxmox_notify`, so should be fine.

> I guess we could do a
>
> 	let config = Config::new(&content, &priv_content)?;
>
> 	Ok((config, config.digest()))
>
> here, but I'm not sure it gains us much?
>
>>
>>> +}
>>> +
>>> +/// Save notification config.
>>> +pub fn save_config(config: Config) -> Result<(), Error> {
>>> +    let (cfg, priv_cfg) = config.write()?;
>>> +    replace_config(NOTIFICATION_CONFIG_PATH, cfg.as_bytes())?;
>>> +    replace_config(NOTIFICATION_PRIV_CONFIG_PATH, priv_cfg.as_bytes())?;
>>
>> the privileged config should probably be saved with higher permission
>> than the general config. consider using `replace_secret_config` here.
>>
>
> Unlike PBS, the notification system will run in the unprivileged process
> in PDM, so that webhook/smtp stuff will not run as root. This means that
> we also need to store the secrets with less strict permissions. We
> already do the same for remotes.cfg and remotes.shadow, so this should
> be fine here as well, I think.
> Of course, we could also use the same approach as in PBS (notification
> code runs as root, secrets are 0600 root:root), but I prefer the
> approach used here.

ack, was not aware that we want to switch to this approach here. but in
hindsight it makes sense why `test_target` wasn't marked as protected.

>>> +/// Add a new gotify endpoint.
>>> +pub fn add_endpoint(
>>> +    endpoint: GotifyConfig,
>>> +    token: String,
>>> +    _rpcenv: &mut dyn RpcEnvironment,
>>> +) -> Result<(), Error> {
>>> +    let _lock = pdm_config::notifications::lock_config()?;
>>> +    let mut config = pdm_config::notifications::config()?;
>>
>> this should probably verify the digest. if you return a config
>> digest as described above this could become:
>>
>>     let mut (config, expected_digest) = domains::config()?;
>>     expected_digest.detect_modification(digest.as_ref())?;
>>
>> assuming that you take the digest as an input parameter to this api call
>> with:
>>
>>     digest: {
>>         optional: true,
>>         type: ConfigDigest,
>>     }
>>
>> and:
>>
>>     digest: Option<ConfigDigest>
>>
>
> We lock the config anyway, so it does not really matter (I think?) if
> somebody else modified some other entity? And if somebody had added an
> entity with the same id, then we fail anyways... So I'm not sure if
> using the digest here gains us anything?
>
> I quickly checked a couple add_* handlers in PBS, we don't
> really check the digest when adding new entities there as well.
>

hm yeah i guess you are right, it wouldn't add too much. guess im just a
little paranoid about modifying on top of an unknown state. but should
probably be fine in these cases.

> Generally, with regards to using the ConfigDigest type: +1
> I think this type was introduced well after the notification API was
> added to PBS, which might be the reason I did not use them there.

yep hence why i think switching to it would make sense now.

>>> +pub fn delete_endpoint(name: String, _rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
>>> +    let _lock = pdm_config::notifications::lock_config()?;
>>> +    let mut config = pdm_config::notifications::config()?;
>>> +    proxmox_notify::api::gotify::delete_gotify_endpoint(&mut config, &name)?;
>>> +
>>
>> this would probably also benefit from checking the config digest.
>>
>
> Same here, I'm not really sure what the benefit would be here?
>
> IMO there are a couple cases to consider for concurrent modifications:
>   - somebody modified entity A, we delete B -> should be fine
>   - somebody modified entity A, we delete A -> does not matter, we want
>     to delete it anyways
>   - we deleted A, somebody else modifies A -> update fails anyways due
>     to the wrong config digest or the entity missing already
>   - both try to delete A -> fails for one of both due to the already
>     deleted entity (HTTP 404)
>
> Did I miss something?
>

see above.

-->8 snip 8<--

Re: [PATCH datacenter-manager v2] Add notifications backend

[Thomas Lamprecht]

On 09/04/2026 14:07, Lukas Wagner wrote:
> This was coordinated with me, I told him that it would be okay to send
> this part early. He can either build on top of this series and include
> more patches (e.g. adding the GUI, notification events, docs, etc.) in
> future versions, or if it makes sense we can also apply certain parts
> early.

I'd strongly recommend to generally send along (basic) integration of
such factoring out work, as else it's basically impossible to review and
see that this can work out without the reviewer having to do the work
themselves. This doesn't have to be a fully fledged out thing for an
initial review, but here some basic notification wouldn't be that much
(and has to be done anyway, as how else was this tested?)

btw. I (and probably others) would also appreciate if such discussion
happens in some shared channel (e.g. zulip).

On 09/04/2026 14:07, Lukas Wagner wrote:
>> this would probably also benefit from checking the config digest.
>>
> Same here, I'm not really sure what the benefit would be here?
> 
> IMO there are a couple cases to consider for concurrent modifications:
>   - somebody modified entity A, we delete B -> should be fine
>   - somebody modified entity A, we delete A -> does not matter, we want
>     to delete it anyways

but we did so with the old config values in mind. The somebody might
have added changed a comment from "test" to "production, don't delete"

The point of digest checks are to ensure an action is executed with
the same information the client saw when making the decision.

>   - we deleted A, somebody else modifies A -> update fails anyways due
>     to the wrong config digest or the entity missing already
>   - both try to delete A -> fails for one of both due to the already
>     deleted entity (HTTP 404)
> 
> Did I miss something?

Another one would be: Somebody deleted A, somebody creates A, we
now delete (another!) A.

Config digest checks are a common thing in most of our stacks for
a reason, they certainly are not the most elaborate/best tool, but
they are simple and protect against any decisions that got wrong due
to out-of-date information. I'd be fine with something better, but
ideally it's some slight variation of the current system, like e.g.
reducing the digest assertion-scope from the whole config to an config
entry (not always useful if there are cross-references in the config),
bigger change can be OK, but naturally needs much more justification
and ensuring that it can be used more than once (ideally everywhere).

Re: [PATCH datacenter-manager v2] Add notifications backend

[Arthur Bied-Charreton]

On Thu, Apr 09, 2026 at 08:26:58PM +0200, Thomas Lamprecht wrote:
> On 09/04/2026 14:07, Lukas Wagner wrote:
> > This was coordinated with me, I told him that it would be okay to send
> > this part early. He can either build on top of this series and include
> > more patches (e.g. adding the GUI, notification events, docs, etc.) in
> > future versions, or if it makes sense we can also apply certain parts
> > early.
> 
> I'd strongly recommend to generally send along (basic) integration of
> such factoring out work, as else it's basically impossible to review and
> see that this can work out without the reviewer having to do the work
> themselves. This doesn't have to be a fully fledged out thing for an
> initial review, but here some basic notification wouldn't be that much
> (and has to be done anyway, as how else was this tested?)
> 
I tested it with curl scripts, I agree that this is not ideal for the
reviewing process. Will send v2 along with integrated notifications.

> btw. I (and probably others) would also appreciate if such discussion
> happens in some shared channel (e.g. zulip).
> 
Noted.
> On 09/04/2026 14:07, Lukas Wagner wrote:
> >> this would probably also benefit from checking the config digest.
> >>
> > Same here, I'm not really sure what the benefit would be here?
> > 
> > IMO there are a couple cases to consider for concurrent modifications:
> >   - somebody modified entity A, we delete B -> should be fine
> >   - somebody modified entity A, we delete A -> does not matter, we want
> >     to delete it anyways
> 
> but we did so with the old config values in mind. The somebody might
> have added changed a comment from "test" to "production, don't delete"
> 
> The point of digest checks are to ensure an action is executed with
> the same information the client saw when making the decision.
> 
> >   - we deleted A, somebody else modifies A -> update fails anyways due
> >     to the wrong config digest or the entity missing already
> >   - both try to delete A -> fails for one of both due to the already
> >     deleted entity (HTTP 404)
> > 
> > Did I miss something?
> 
> Another one would be: Somebody deleted A, somebody creates A, we
> now delete (another!) A.
> 
> Config digest checks are a common thing in most of our stacks for
> a reason, they certainly are not the most elaborate/best tool, but
> they are simple and protect against any decisions that got wrong due
> to out-of-date information. I'd be fine with something better, but
> ideally it's some slight variation of the current system, like e.g.
> reducing the digest assertion-scope from the whole config to an config
> entry (not always useful if there are cross-references in the config),
> bigger change can be OK, but naturally needs much more justification
> and ensuring that it can be used more than once (ideally everywhere).

Thanks a lot for the context, I will integrate whole config digest 
checks in the handlers in v2.