From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id A70F71FF0E0 for ; Thu, 06 Aug 2026 15:09:39 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 766A121536; Thu, 06 Aug 2026 15:09:39 +0200 (CEST) Date: Thu, 06 Aug 2026 15:09:27 +0200 From: Fabian =?iso-8859-1?q?Gr=FCnbichler?= Subject: Re: [PATCH proxmox-backup v3 06/16] config/server/api: add certificate renewal logic including notifications To: pbs-devel@lists.proxmox.com, Shannon Sterz References: <20260805155308.519896-2-s.sterz@proxmox.com> <20260805155308.519896-8-s.sterz@proxmox.com> In-Reply-To: <20260805155308.519896-8-s.sterz@proxmox.com> MIME-Version: 1.0 User-Agent: astroid/0.17.0 (https://github.com/astroidmail/astroid) Message-Id: <1786020761.s6q0zxwz9u.astroid@yuna.none> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1786021755243 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.114 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_LOW -0.7 Sender listed at https://www.dnswl.org/, low trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: S4HYNKVQFPURP4XS4FZBKVH472JPTVGJ X-Message-ID-Hash: S4HYNKVQFPURP4XS4FZBKVH472JPTVGJ X-MailFrom: f.gruenbichler@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On August 5, 2026 5:52 pm, Shannon Sterz wrote: > the daily-update service is used to check whether a self-signed > certificate is in use and renews it if it would expire within the next > 15 days. it will also send out reminder notifications starting 15 days > before a certificate could be renewed. >=20 > a self-signed certificate is detected by parsing the issuer field of > the certificate. if it aligns with how self-signed certificates are > created by this instance of proxmox backup server, it will be deemed > self-signed. >=20 > 15 days was chosen as that should give a reasonable trade-off between > not failing if the daily-update service can't run on the specific day > that the certificate would expire and not refreshing the certificate > too often. note that 15 days before expiry corresponds to Let's > Encrypt's recommendation for when to update new certificates that are > valid for 45 days: >=20 >> Acceptable behavior includes renewing certificates at approximately >> two thirds of the way through the current certificate=E2=80=99s lifetime= . >> >> - https://letsencrypt.org/2025/12/02/from-90-to-45#action-required >=20 > 15 days before expiry is about two thirds of the lifetime of a > certificate that lasts for 45 days. >=20 > Signed-off-by: Shannon Sterz > --- >=20 > Notes: > currently this will send out a notification every time the service is > run in the 15 days between a renewal being imminent and not being > renewed yet. this is probably excessive and we should limit the > notifications here to only go out once per week (or similar). > =20 > i chose 15 days instead of two thirds of ten years, because for a lon= g > lasting certificate like that, trying to refresh it starting from 3.3 > years before it expires seems excessive to me. I think some sort of lifetime-based-but-clamped approach might make the most sense? e.g.., not more than a third of the lifetime remaining, but at least 2 days before, and at most half a year before expiry? ideally we could share the expiry logic between ACME and self-signed certificates, in particular if we split the "is this self-signed" logic from "does it expire soon" check? >=20 > debian/proxmox-backup-server.install | 6 ++ > src/api2/node/certificates.rs | 9 ++- > src/bin/proxmox-daily-update.rs | 44 +++++++++++- > src/config/mod.rs | 4 +- > src/server/notifications/mod.rs | 64 +++++++++++++++-- > src/server/notifications/template_data.rs | 22 ++++++ > templates/Makefile | 68 ++++++++++--------- > templates/default/cert-err-body.txt.hbs | 7 ++ > templates/default/cert-err-subject.txt.hbs | 1 + > templates/default/cert-refresh-body.txt.hbs | 8 +++ > .../default/cert-refresh-subject.txt.hbs | 1 + > .../cert-upcoming-refresh-body.txt.hbs | 10 +++ > .../cert-upcoming-refresh-subject.txt.hbs | 1 + > 13 files changed, 207 insertions(+), 38 deletions(-) > create mode 100644 templates/default/cert-err-body.txt.hbs > create mode 100644 templates/default/cert-err-subject.txt.hbs > create mode 100644 templates/default/cert-refresh-body.txt.hbs > create mode 100644 templates/default/cert-refresh-subject.txt.hbs > create mode 100644 templates/default/cert-upcoming-refresh-body.txt.hbs > create mode 100644 templates/default/cert-upcoming-refresh-subject.txt.h= bs >=20 > diff --git a/debian/proxmox-backup-server.install b/debian/proxmox-backup= -server.install > index 1f1d9b601..c485b28a9 100644 > --- a/debian/proxmox-backup-server.install > +++ b/debian/proxmox-backup-server.install > @@ -45,6 +45,12 @@ usr/share/man/man5/user.cfg.5 > usr/share/man/man5/verification.cfg.5 > usr/share/proxmox-backup/templates/default/acme-err-body.txt.hbs > usr/share/proxmox-backup/templates/default/acme-err-subject.txt.hbs > +usr/share/proxmox-backup/templates/default/cert-err-body.txt.hbs > +usr/share/proxmox-backup/templates/default/cert-err-subject.txt.hbs > +usr/share/proxmox-backup/templates/default/cert-refresh-body.txt.hbs > +usr/share/proxmox-backup/templates/default/cert-refresh-subject.txt.hbs > +usr/share/proxmox-backup/templates/default/cert-upcoming-refresh-body.tx= t.hbs > +usr/share/proxmox-backup/templates/default/cert-upcoming-refresh-subject= .txt.hbs > usr/share/proxmox-backup/templates/default/gc-err-body.txt.hbs > usr/share/proxmox-backup/templates/default/gc-err-subject.txt.hbs > usr/share/proxmox-backup/templates/default/gc-ok-body.txt.hbs > diff --git a/src/api2/node/certificates.rs b/src/api2/node/certificates.r= s > index 3df05b020..6075b5bd8 100644 > --- a/src/api2/node/certificates.rs > +++ b/src/api2/node/certificates.rs > @@ -96,7 +96,7 @@ pub struct CertificateInfo { > pub fingerprint: Option, > } > =20 > -fn get_certificate_pem() -> Result { > +pub fn get_certificate_pem() -> Result { > let cert_path =3D configdir!("/proxy.pem"); > let cert_pem =3D proxmox_sys::fs::file_get_contents(cert_path)?; > String::from_utf8(cert_pem) > @@ -354,6 +354,13 @@ pub fn check_renewal_needed() -> Result<(bool, i64),= Error> { > Ok((expires_soon, lead / SECONDS_PER_DAY)) > } > =20 > +/// Renews the self-signed certificate. The caller needs to make sure th= e current certificate is > +/// really a self-signed certificate and not an ACME or custom certifica= te. > +pub async fn renew_self_signed_cert() -> Result<(), Error> { > + crate::config::update_self_signed_cert(true)?; > + crate::server::reload_proxy_certificate().await > +} > + > fn spawn_certificate_worker( > name: &'static str, > force: bool, > diff --git a/src/bin/proxmox-daily-update.rs b/src/bin/proxmox-daily-upda= te.rs > index 597722497..200d911e3 100644 > --- a/src/bin/proxmox-daily-update.rs > +++ b/src/bin/proxmox-daily-update.rs > @@ -1,12 +1,15 @@ > use anyhow::Error; > use serde_json::json; > =20 > +use proxmox_backup::server::notifications::{ > + send_self_signed_renewal_notification, send_upcoming_self_signed_ren= ewal_notification, > +}; > use proxmox_notify::context::pbs::PBS_CONTEXT; > use proxmox_router::{ApiHandler, RpcEnvironment, cli::*}; > use proxmox_subscription::SubscriptionStatus; > =20 > use pbs_buildcfg::configdir; > -use proxmox_backup::api2; > +use proxmox_backup::{api2, config}; > =20 > async fn wait_for_local_worker(upid_str: &str) -> Result<(), Error> { > let upid: pbs_api_types::UPID =3D upid_str.parse()?; > @@ -60,6 +63,10 @@ async fn do_update(rpcenv: &mut dyn RpcEnvironment) ->= Result<(), Error> { > log::error!("error checking certificates: {err}"); > } > =20 > + if let Err(err) =3D renew_self_signed_certificate().await { > + log::error!("error checking self-signed certificate renewal: {er= r:#}"); > + } > + > // TODO: cleanup tasks like in PVE? > =20 > Ok(()) > @@ -89,6 +96,41 @@ async fn check_acme_certificates(rpcenv: &mut dyn RpcE= nvironment) -> Result<(), > Ok(()) > } > =20 > +async fn renew_self_signed_certificate() -> Result<(), Error> { > + let resolv_conf =3D crate::api2::node::dns::read_etc_resolv_conf()?; > + let pem =3D api2::node::certificates::get_certificate_pem()?; > + > + let days =3D match proxmox_tls_certificates::self_signed_cert_expire= s_in( > + config::PRODUCT_NAME, > + proxmox_sys::nodename(), > + resolv_conf["search"].as_str(), > + proxmox_tls_certificates::CertificateInfo::from_pem("proxy.pem",= pem.as_bytes())?, > + )? { > + None =3D> { > + log::debug!("Certificate is not self-signed, nothing to do."= ); > + return Ok(()); > + } > + Some(days) =3D> days as i64, > + }; > + > + if days <=3D 15 { > + log::info!("Certificate expires within 15 days, renewing certifi= cate..."); > + let res =3D api2::node::certificates::renew_self_signed_cert().a= wait; > + > + if let Err(e) =3D &res { > + log::warn!("Could not renew self-signed certificate - {e:#}"= ); > + } > + > + send_self_signed_renewal_notification(&res)?; > + } else if days <=3D 30 { > + log::info!("Certificate expires within 30 days, notify about ren= ewal."); > + let earliest_renewal =3D proxmox_time::epoch_i64() + (days - 15)= * 24 * 60 * 60; > + send_upcoming_self_signed_renewal_notification(earliest_renewal)= ?; > + } > + > + Ok(()) > +} > + > async fn run(rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> { > proxmox_product_config::init(pbs_config::backup_user()?, pbs_config:= :priv_user()?); > proxmox_rest_server::init_worker_tasks( > diff --git a/src/config/mod.rs b/src/config/mod.rs > index bbca5a9af..57a5c7a06 100644 > --- a/src/config/mod.rs > +++ b/src/config/mod.rs > @@ -15,6 +15,8 @@ use pbs_buildcfg::{self, configdir}; > =20 > pub mod tfa; > =20 > +pub const PRODUCT_NAME: &str =3D "Proxmox Backup Server"; > + > /// Check configuration directory permissions > /// > /// For security reasons, we want to make sure they are set correctly: > @@ -90,7 +92,7 @@ pub fn update_self_signed_cert(force: bool) -> Result<(= ), Error> { > let resolv_conf =3D crate::api2::node::dns::read_etc_resolv_conf()?; > =20 > let (priv_key, cert) =3D proxmox_tls_certificates::create_self_signe= d_cert( > - "Proxmox Backup Server", > + PRODUCT_NAME, > proxmox_sys::nodename(), > resolv_conf["search"].as_str(), > None, > diff --git a/src/server/notifications/mod.rs b/src/server/notifications/m= od.rs > index 181f12cbc..487975582 100644 > --- a/src/server/notifications/mod.rs > +++ b/src/server/notifications/mod.rs > @@ -10,6 +10,7 @@ use proxmox_notify::context::pbs::PBS_CONTEXT; > use proxmox_schema::ApiType; > use proxmox_sys::fs::{CreateOptions, create_path}; > =20 > +use crate::server::notifications::template_data::CertUpcomingRefreshTemp= lateData; > use crate::tape::TapeNotificationMode; > use pbs_api_types::{ > APTUpdateInfo, DataStoreConfig, DatastoreNotify, GarbageCollectionSt= atus, NotificationMode, > @@ -23,10 +24,10 @@ const SPOOL_DIR: &str =3D concatcp!(pbs_buildcfg::PRO= XMOX_BACKUP_STATE_DIR, "/noti > mod template_data; > =20 > use template_data::{ > - AcmeErrTemplateData, CommonData, DatastoreThresholdExceededTemplateD= ata, GcErrTemplateData, > - GcOkTemplateData, PackageUpdatesTemplateData, PruneErrTemplateData, = PruneOkTemplateData, > - SyncErrTemplateData, SyncOkTemplateData, TapeBackupErrTemplateData, = TapeBackupOkTemplateData, > - TapeLoadTemplateData, VerifyErrTemplateData, VerifyOkTemplateData, > + AcmeErrTemplateData, CertErrTemplateData, CommonData, DatastoreThres= holdExceededTemplateData, > + GcErrTemplateData, GcOkTemplateData, PackageUpdatesTemplateData, Pru= neErrTemplateData, > + PruneOkTemplateData, SyncErrTemplateData, SyncOkTemplateData, TapeBa= ckupErrTemplateData, > + TapeBackupOkTemplateData, TapeLoadTemplateData, VerifyErrTemplateDat= a, VerifyOkTemplateData, > }; > =20 > /// Initialize the notification system by setting context in proxmox_not= ify > @@ -575,6 +576,61 @@ pub fn send_certificate_renewal_mail(result: &Result= <(), Error>) -> Result<(), E > Ok(()) > } > =20 > +/// Send email for upcoming self signed renewal. > +/// > +/// * `earliest_renewal`: A Unix timestamp specifying the earliest a cer= tificate may be renewed. > +pub fn send_upcoming_self_signed_renewal_notification(earliest_renewal: = i64) -> Result<(), Error> { > + let metadata =3D HashMap::from([ > + ("hostname".into(), proxmox_sys::nodename().into()), > + ("type".into(), "cert".into()), > + ]); > + > + let notification =3D Notification::from_template( > + Severity::Info, > + "cert-upcoming-refresh", > + serde_json::to_value(CertUpcomingRefreshTemplateData { > + common: CommonData::new(), > + earliest_renewal, > + })?, > + metadata, > + ); > + > + send_notification(notification)?; > + Ok(()) > +} > + > +/// Send email renewed self-signed certificate > +pub fn send_self_signed_renewal_notification(result: &Result<(), Error>)= -> Result<(), Error> { > + let metadata =3D HashMap::from([ > + ("hostname".into(), proxmox_sys::nodename().into()), > + ("type".into(), "cert-renewal".into()), > + ]); > + > + let notification =3D match result { > + Err(e) =3D> { > + let template_data =3D CertErrTemplateData { > + common: CommonData::new(), > + error: format!("{e:#}"), > + }; > + > + Notification::from_template( > + Severity::Error, > + "cert-err", > + serde_json::to_value(template_data)?, > + metadata, > + ) > + } > + _ =3D> Notification::from_template( > + Severity::Notice, > + "cert-refresh", > + serde_json::to_value(CommonData::new())?, > + metadata, > + ), > + }; > + > + send_notification(notification) > +} > + > /// Send notification if datastore values are exceeding the set threshol= d limit. > pub fn send_datastore_threshold_exceeded( > datastore: &str, > diff --git a/src/server/notifications/template_data.rs b/src/server/notif= ications/template_data.rs > index e10215bdc..649d8b127 100644 > --- a/src/server/notifications/template_data.rs > +++ b/src/server/notifications/template_data.rs > @@ -145,6 +145,28 @@ pub struct AcmeErrTemplateData { > pub error: String, > } > =20 > +/// Template data for the cert-err template. > +#[derive(Serialize)] > +#[serde(rename_all =3D "kebab-case")] > +pub struct CertErrTemplateData { > + /// Common properties. > + #[serde(flatten)] > + pub common: CommonData, > + /// The error that occurred when trying to request the certificate. > + pub error: String, > +} > + > +/// Template data for the cert-upcoming-refresh template. > +#[derive(Serialize)] > +#[serde(rename_all =3D "kebab-case")] > +pub struct CertUpcomingRefreshTemplateData { > + /// Common properties. > + #[serde(flatten)] > + pub common: CommonData, > + /// A Unix timestamp representing the earliest point in time a certi= ficate may be renewed. > + pub earliest_renewal: i64, > +} > + > #[derive(Serialize)] > #[serde(rename_all =3D "kebab-case")] > /// A single package which can be upgraded. > diff --git a/templates/Makefile b/templates/Makefile > index 8a4586d78..f086b927b 100644 > --- a/templates/Makefile > +++ b/templates/Makefile > @@ -1,36 +1,42 @@ > include ../defines.mk > =20 > -NOTIFICATION_TEMPLATES=3D \ > - default/acme-err-body.txt.hbs \ > - default/acme-err-subject.txt.hbs \ > - default/gc-err-body.txt.hbs \ > - default/gc-ok-body.txt.hbs \ > - default/gc-err-subject.txt.hbs \ > - default/gc-ok-subject.txt.hbs \ > - default/package-updates-body.txt.hbs \ > - default/package-updates-subject.txt.hbs \ > - default/prune-err-body.txt.hbs \ > - default/prune-ok-body.txt.hbs \ > - default/prune-err-subject.txt.hbs \ > - default/prune-ok-subject.txt.hbs \ > - default/sync-err-body.txt.hbs \ > - default/sync-ok-body.txt.hbs \ > - default/sync-err-subject.txt.hbs \ > - default/sync-ok-subject.txt.hbs \ > - default/tape-backup-err-body.txt.hbs \ > - default/tape-backup-err-subject.txt.hbs \ > - default/tape-backup-ok-body.txt.hbs \ > - default/tape-backup-ok-subject.txt.hbs \ > - default/tape-load-body.txt.hbs \ > - default/tape-load-subject.txt.hbs \ > - default/test-body.txt.hbs \ > - default/test-subject.txt.hbs \ > - default/thresholds-exceeded-body.txt.hbs \ > - default/thresholds-exceeded-subject.txt.hbs \ > - default/verify-err-body.txt.hbs \ > - default/verify-ok-body.txt.hbs \ > - default/verify-err-subject.txt.hbs \ > - default/verify-ok-subject.txt.hbs \ > +NOTIFICATION_TEMPLATES=3D \ > + default/acme-err-body.txt.hbs \ > + default/acme-err-subject.txt.hbs \ > + default/cert-err-body.txt.hbs \ > + default/cert-err-subject.txt.hbs \ > + default/cert-refresh-body.txt.hbs \ > + default/cert-refresh-subject.txt.hbs \ > + default/cert-upcoming-refresh-body.txt.hbs \ > + default/cert-upcoming-refresh-subject.txt.hbs \ > + default/gc-err-body.txt.hbs \ > + default/gc-ok-body.txt.hbs \ > + default/gc-err-subject.txt.hbs \ > + default/gc-ok-subject.txt.hbs \ > + default/package-updates-body.txt.hbs \ > + default/package-updates-subject.txt.hbs \ > + default/prune-err-body.txt.hbs \ > + default/prune-ok-body.txt.hbs \ > + default/prune-err-subject.txt.hbs \ > + default/prune-ok-subject.txt.hbs \ > + default/sync-err-body.txt.hbs \ > + default/sync-ok-body.txt.hbs \ > + default/sync-err-subject.txt.hbs \ > + default/sync-ok-subject.txt.hbs \ > + default/tape-backup-err-body.txt.hbs \ > + default/tape-backup-err-subject.txt.hbs \ > + default/tape-backup-ok-body.txt.hbs \ > + default/tape-backup-ok-subject.txt.hbs \ > + default/tape-load-body.txt.hbs \ > + default/tape-load-subject.txt.hbs \ > + default/test-body.txt.hbs \ > + default/test-subject.txt.hbs \ > + default/thresholds-exceeded-body.txt.hbs \ > + default/thresholds-exceeded-subject.txt.hbs \ > + default/verify-err-body.txt.hbs \ > + default/verify-ok-body.txt.hbs \ > + default/verify-err-subject.txt.hbs \ > + default/verify-ok-subject.txt.hbs \ > =20 > all: > =20 > diff --git a/templates/default/cert-err-body.txt.hbs b/templates/default/= cert-err-body.txt.hbs > new file mode 100644 > index 000000000..27f444517 > --- /dev/null > +++ b/templates/default/cert-err-body.txt.hbs > @@ -0,0 +1,7 @@ > +Proxmox Backup Server was not able to renew a self-signed TLS certificat= e. > + > +Error: {{error}} > + > +Please visit the web interface for further details: > + > +<{{base-url}}/#pbsCertificateConfiguration> > diff --git a/templates/default/cert-err-subject.txt.hbs b/templates/defau= lt/cert-err-subject.txt.hbs > new file mode 100644 > index 000000000..7b8f2a20e > --- /dev/null > +++ b/templates/default/cert-err-subject.txt.hbs > @@ -0,0 +1 @@ > +Could not renew self-signed certificate > diff --git a/templates/default/cert-refresh-body.txt.hbs b/templates/defa= ult/cert-refresh-body.txt.hbs > new file mode 100644 > index 000000000..608ba30c0 > --- /dev/null > +++ b/templates/default/cert-refresh-body.txt.hbs > @@ -0,0 +1,8 @@ > +Proxmox Backup Server has refreshed its self-signed TLS certificate. > + > +The new certificate is now active. Please update any clients relying on = the > +certificate fingerprint to verify their connection to the server. > + > +Please visit the web interface for further details: > + > +<{{base-url}}/#pbsCertificateConfiguration> > diff --git a/templates/default/cert-refresh-subject.txt.hbs b/templates/d= efault/cert-refresh-subject.txt.hbs > new file mode 100644 > index 000000000..e5ebb4074 > --- /dev/null > +++ b/templates/default/cert-refresh-subject.txt.hbs > @@ -0,0 +1 @@ > +Self-signed certificate has been refreshed > diff --git a/templates/default/cert-upcoming-refresh-body.txt.hbs b/templ= ates/default/cert-upcoming-refresh-body.txt.hbs > new file mode 100644 > index 000000000..256fcda01 > --- /dev/null > +++ b/templates/default/cert-upcoming-refresh-body.txt.hbs > @@ -0,0 +1,10 @@ > +Proxmox Backup Server will refresh its TLS certificate within the next 3= 0 days. > + > +The earliest the certificate may be renewed is {{ timestamp earliest-ren= ewal }}. > +If you rely on the certificate's fingerprint to verify TLS sessions betw= een the > +server and a client, please update the fingerprint once the certificate = has > +been updated. Otherwise, no action is required. > + > +Please visit the web interface for further details: > + > +<{{base-url}}/#pbsCertificateConfiguration> > diff --git a/templates/default/cert-upcoming-refresh-subject.txt.hbs b/te= mplates/default/cert-upcoming-refresh-subject.txt.hbs > new file mode 100644 > index 000000000..27c762b2d > --- /dev/null > +++ b/templates/default/cert-upcoming-refresh-subject.txt.hbs > @@ -0,0 +1 @@ > +Self-signed certificate is about to be refreshed > --=20 > 2.47.3 >=20 >=20 >=20 >=20 >=20 >=20