Re: [PATCH cluster/common/datacenter-manager/manager/proxmox v2 00/16] TLS Certificate Staging

["Fabian Grünbichler" <f.gruenbichler@proxmox.com>]

On August 5, 2026 3:18 pm, Shannon Sterz wrote:
> the aim of this series is to allow clients to automatically adapt to regular
> certificate rotation. the top-level overview of the mechanism proposed here is
> as follows:

I like the principal approach of this. left some comments that are
hopefully actionable. we should also do the same thing for PBS, which
means we also need handling code for staging fingerprints in the PBS
client(s).

I think most of the perl certificate handling code can probably move to
the proxmox-tls-certificates crate you want to introduce. if that
exposes the right set of helpers, then we hopefully only have a single
code base for all of the lower level certificate handling (at some
point).

> - hosts that rotate their certificate create a new certificate at the earliest
>   four weeks before their current certificate expires. this certificate is
>   considered as "staged" up until it becomes actively used.
> - clients can query a host for a staged certificate at any moment, the host
>   will provide information such as the fingerprint for the active and staged
>   certificate(s).
> - at the earliest two weeks before their current certificate expires, hosts may
>   start using the "staged" certificate. the two week window is needed to give
>   clients enough time to query a potential staged certificate.
> - clients, that use fingerprints to validate a TLS certificate, should discard
>   the previously used fingerprint and update to the new certificate's
>   fingerprint (the previously staged certificate) as soon as they detect its
>   usage. connections trying to authenticate themselves with the old certificate
>   should be rejected at this point.
> 
> this series implements the host part of this mechanism for pve 9 and pdm. the
> first three patches in the series are intended for pve and implement the
> staging mechanism. they also make it a little easier to query the certificate
> of a node when we don't know the node name specifically.
> 
> the next few patches improve how fingerprints are handled for pdm. they also
> add the certificate info endpoint to the pve client. specifically the following
> improvements are provided:
> 
> * if a fingerprint of a remote does not match, but pdm-client is in interactive
>   mode, allow a user to accept the updated fingerprint then and there. this
>   better matches the behaviour in interactive mode of connecting to a
>   non-trusted node (patch 6).
> * report mismatching fingerprints as untrusted when probing a remote and
>   improve how the ui handles such situations by adding more context (patches
>   8-10)
> 
> the remaining patches mostly prepare and then implement the rotation mechanism
> within pdm. pdm will query pve remote nodes once every twelve hours to see if a
> new staged certificate becomes available. if a new fingerprint is encountered,
> it will be stored in the remotes.cfg. once a staged fingerprint is encountered,
> it will replace the active fingerprint.
> 
> How to Test
> -----------
> 
> the easiest way is probably to force pve to rotate and stage certificates by
> setting a date with `date --set` that's far enough in the future to trigger the
> action and then running `pveupdate`. to force pdm to query its remotes, it's
> easiest to run `systemctl restart proxmox-datacenter-api.service`. the daemon
> will execute the task query its remotes once on start.
> 
> How to Apply & Bump
> -------------------
> 
> the first patch for pve-manager (03/16) depends on the changes for pve-cluster
> (01/16) and pve-common (02/16). the second pve-manager patch can be applied
> independently. note that Elias has sent patches [1] that would address the same
> issue as patch (04/16), so that patch can be dropped in case Elias' series
> makes it in first.
> 
> the patches for proxmox-datacenter-manager can all be applied independently,
> with the exception of the last one (16/16), which needs the patch for the
> pve-api-types (05/16) to be applied and bumped.
> 
> Future Work
> -----------
> 
> 1. pbs remotes currently do not rotate their certificates. a series that is as
> of yet not applied would add such a mechanism to pbs too [2]. for now pbs
> remotes are ignored by the staged certificates mechanism for the most part.
> 
> 2. the `Fingerprint` type should be replaced by one from a shared proxmox-*
> crate. Dominik's series for unifying tls callbacks adds such a type to
> proxmox-http [3]. i'll adapt this series depending on how things are applied.
> 
> 3. backporting of the pve patches to the bookworm branch probably makes sense
> to improve compatibility. i'll send such patches once this series is approved.
> this may have been more prescient when this series was first submitted. since
> pve 8 is eol by now (or tomorrow as of sending this), this may no longer apply.
> 
> 4. somewhat orthogonal to this series: the mechanism outlined in the notes of
> patch 15 would probably improve adding tasks to pdm. talked about this with
> Lukas a little and he agrees. possibly in combination with a message passing
> mechanism.
> 
> 5. i'll add documentation to this series once the general mechanism is
> deemed appropriate or if it's applied earlier in a follow-up.
> 
> Changelog
> ---------
> 
> * v1: https://lore.proxmox.com/all/20260731091655.93282-1-s.sterz@proxmox.com/
> 
>     changes since v1:
> 
>     + always return leaf certificate fingerprint when probing remotes
>     + clean up an issue where the staged certificate could be regenerated once
>       a day during the staging period
>     + clean up usages of `tokio::spawn` that did blocking io with
>       `tokio::task::spawn_blocking`
>     + fix an issue where the ca could have been rotated between staging a
>       certificate and using it.
>     + clean up stale staged certificates properly
>     + other smaller clean ups (more correct phrasings etc).
> 
> * rfc: https://lore.proxmox.com/all/20260611120327.257523-1-s.sterz@proxmox.com/
> 
>     changes since the rfc:
> 
>     + dropped a patch adjusting the tls verify callback in proxmox-client. it
>       should instead be replaced by Dominik's implementation of a unified
>       callback once that's applied
>     + dropped a similar patch for proxmox-datacenter-client for the same
>       reason.
>     + fixed an issue where a certificate chain could be validated incorrectly
>     + rebased on current master for all repos
> 
> [1]: https://lore.proxmox.com/all/20260714145027.53038-1-e.huhsovitz@proxmox.com/
> [2]: https://lore.proxmox.com/all/20260730133158.418015-1-s.sterz@proxmox.com/
> [3]: https://lore.proxmox.com/all/20260701103120.1593265-6-d.csapak@proxmox.com/
> 
> 
> pve-cluster:
> 
> Shannon Sterz (1):
>   setup: allow caller to provide the certificate filename
> 
>  src/PVE/Cluster/Setup.pm | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> 
> pve-common:
> 
> Shannon Sterz (1):
>   certificate: add helper to verify that a certificate was signed by a
>     ca
> 
>  src/PVE/Certificate.pm | 46 ++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 46 insertions(+)
> 
> 
> pve-manager:
> 
> Shannon Sterz (2):
>   bin/api: add a new staged certificate when renewing self-signed cert
>   api: certificates: if node parameter is 'localhost' return local certs
> 
>  PVE/API2/Certificates.pm | 10 ++++--
>  PVE/CertHelpers.pm       |  6 ++++
>  bin/pveupdate            | 72 +++++++++++++++++++++++++++++++++-------
>  3 files changed, 74 insertions(+), 14 deletions(-)
> 
> 
> proxmox:
> 
> Shannon Sterz (1):
>   pve-api-types: expose certificates info endpoint
> 
>  pve-api-types/Cargo.toml            |  1 +
>  pve-api-types/debian/control        |  2 ++
>  pve-api-types/generate.pl           |  3 +++
>  pve-api-types/src/generated/code.rs | 15 ++++++++++++++-
>  pve-api-types/src/types/mod.rs      |  1 +
>  5 files changed, 21 insertions(+), 1 deletion(-)
> 
> 
> proxmox-datacenter-manager:
> 
> Shannon Sterz (11):
>   client: allow users to update a changed fingerprint interactively
>   cli/api-types: move Fingerprint to common api type crate
>   server: connection: report mismatching fingerprint as untrusted on
>     probe
>   ui: wizard: add context if a provided fingerprint did not match remote
>   ui: wizard: nodes page: always update fingerprints on user
>     confirmation
>   pdm-api-types: implement ApiType for Fingerprint
>   pdm-api-types: add staged_fingerprints field to NodeUrl
>   server: remotes: lock remotes config when updating it
>   server: connection: rotate in staged fingerprints when encountering
>     them
>   server: api: tasks: move `spawn_aborted_on_shutdown()` to super module
>   server: bin: api: tasks: add task to discover new staged certificates
> 
>  cli/client/src/env/fingerprint_cache.rs       |  90 +--------
>  cli/client/src/env/mod.rs                     |   2 +-
>  lib/pdm-api-types/Cargo.toml                  |   1 +
>  lib/pdm-api-types/src/fingerprint.rs          |  84 ++++++++
>  lib/pdm-api-types/src/lib.rs                  |   3 +
>  lib/pdm-api-types/src/remotes.rs              |  15 +-
>  server/src/api/pbs/mod.rs                     |   2 +
>  server/src/api/pve/mod.rs                     |   3 +
>  server/src/api/remotes/mod.rs                 |  36 +++-
>  server/src/bin/proxmox-datacenter-api/main.rs |   1 +
>  .../tasks/ceph_detection.rs                   |  18 +-
>  .../bin/proxmox-datacenter-api/tasks/mod.rs   |  17 ++
>  .../tasks/remote_staged_fingerprints.rs       | 156 +++++++++++++++
>  server/src/connection.rs                      | 179 +++++++++++++++---
>  ui/src/remotes/config.rs                      |   1 +
>  ui/src/remotes/node_url_list.rs               |   1 +
>  ui/src/remotes/wizard_page_connect.rs         |  26 ++-
>  ui/src/remotes/wizard_page_info.rs            |   1 +
>  ui/src/remotes/wizard_page_nodes.rs           |  40 +++-
>  19 files changed, 544 insertions(+), 132 deletions(-)
>  create mode 100644 lib/pdm-api-types/src/fingerprint.rs
>  create mode 100644 server/src/bin/proxmox-datacenter-api/tasks/remote_staged_fingerprints.rs
> 
> 
> Summary over all repositories:
>   29 files changed, 687 insertions(+), 149 deletions(-)
> 
> -- 
> Generated by murpp 0.12.0
> 
> 
> 
> 
>