Superseded: Re: [PATCH cluster/datacenter-manager/manager/proxmox 00/15] TLS Certificate Staging

["Shannon Sterz" <s.sterz@proxmox.com>]

Superseded-by: https://lore.proxmox.com/pdm-devel/20260805131838.254723-2-s.sterz@proxmox.com/

On Fri Jul 31, 2026 at 11:16 AM CEST, Shannon Sterz wrote:
> the aim of this series is to allow clients to automatically adapt to regular
> certificate rotation. the top-level overview of the mechanism proposed here is
> as follows:
>
> - hosts that rotate their certificate create a new certificate at the earliest
>   four weeks before their current certificate expires. this certificate is
>   considered as "staged" up until it becomes actively used.
> - clients can query a host for a staged certificate at any moment, the host
>   will provide information such as the fingerprint for the active and staged
>   certificate(s).
> - at the earliest two weeks before their current certificate expires, hosts may
>   start using the "staged" certificate. the two week window is needed to give
>   clients enough time to query a potential staged certificate.
> - clients, that use fingerprints to validate a TLS certificate, should discard
>   the previously used fingerprint and update to the new certificate's
>   fingerprint (the previously staged certificate) as soon as they detect its
>   usage. connections trying to authenticate themselves with the old Certificate
>   should be rejected at this point.
>
> this series implements the host part of this mechanism for pve 9 and pdm. the
> first three patches in the series are intended for pve and implement the
> staging mechanism. they also make it a little easier to query the certificate
> of a node when we don't know the node name specifically.
>
> the next few patches improve how fingerprints are handled for pdm. they also
> add the certificate info endpoint to the pve client. specifically the following
> improvements are provided:
>
> * if a fingerprint of a remote does not match, but pdm-client is in interactive
>   mode, allow a user to accept the updated fingerprint then and there. this
>   better matches the behaviour in interactive mode of connecting to a
>   non-trusted node (patch 5).
> * report mismatching fingerprints as untrusted when probing a remote and
>   improve how the ui handles such situations by adding more context (patches
>   7-9)
>
> the remaining patches mostly prepare and then implement the rotation mechanism
> within pdm. pdm will query pve remote nodes once every twelve hours to see if a
> new staged certificate becomes available. if a new fingerprint is encountered,
> it will be stored in the remotes.cfg. once a staged fingerprint is encountered,
> it will replace the active fingerprint.
>
> How to Test
> -----------
>
> the easiest way is probably to force pve to rotate and stage certificates by
> setting a date with `date --set` that's far enough in the future to trigger the
> action and then running `pveupdate`. to force pdm to query its remotes, it's
> easiest to run `systemctl restart proxmox-datacenter-api.service`. the daemon
> will execute the task query its remotes once on start.
>
> How to Apply & Bump
> -------------------
>
> the first patch for pve-manager (02/15) depends on the changes for pve-cluster
> (01/15). the second pve-manager patch can be applied independently. note that
> Elias' has send a patches [1] that would address the same issue as patch
> (03/15), so that patch can be dropped in case elias's series makes it in first.
>
> the patches for proxmox-datacenter-manager can all be applied independently,
> with the exception of the last one (15/15), which needs the patch for the
> pve-api-types (04/15) to be applied and bumped.
>
> Future Work
> -----------
>
> 1. pbs remotes currently do not rotate their certificates. a series that is as
> of yet not applied would add such a mechanism to pbs too [1]. for now pbs
> remotes are ignored by the staged certificates mechanism for the most part.
>
> 2. the `Fingerprint` type should be replaced by one from a shared proxmox-*
> crate. Dominik's series for unifying tls callbacks adds such a type to
> proxmox-http [2]. i'll adapt this series depending on how things are applied.
>
> 3. backporting of the pve patches to the bookworm branch probably makes sense
> to improve compatibility. i'll send such patches once this series is. this may
> have been more prescient  when this series was first submitted. since pve 8 is
> eol by now (or tomorrow as of sending this), this may no longer apply.
>
> 4. somewhat orthogonal to this series: the mechanism outlined in the notes of
> patch 16 would probably improve adding tasks to pdm.
>
> Changelog
> ---------
>
> * rfc: https://lore.proxmox.com/all/20260611120327.257523-1-s.sterz@proxmox.com/
>
>     changes since the rfc:
>
>     + dropped a patch adjust the tls verify callback in proxmox-client. it
>       should instead be replaced by Dominik's implementation of a unified
>       callback once that's applied
>     + dropped a similar patch for proxmox-datacenter-client for the same
>       reason.
>     + fixed an issues where a certificate chain could be validated incorrectly
>     + rebased on current master for all repos
>
> [1]: https://lore.proxmox.com/all/20260714145027.53038-1-e.huhsovitz@proxmox.com/
> [2]: https://lore.proxmox.com/all/20260730133158.418015-1-s.sterz@proxmox.com/
> [3]: https://lore.proxmox.com/all/20260701103120.1593265-6-d.csapak@proxmox.com/
>
>
> pve-cluster:
>
> Shannon Sterz (1):
>   setup: allow caller to provide the certificate filename
>
>  src/PVE/Cluster/Setup.pm | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
>
> pve-manager:
>
> Shannon Sterz (2):
>   bin/api: add a new staged certificate when renewing self-signed cert
>   api: certificates: if node parameter is 'localhost' return local certs
>
>  PVE/API2/Certificates.pm | 10 +++++++--
>  PVE/CertHelpers.pm       |  6 ++++++
>  bin/pveupdate            | 44 ++++++++++++++++++++++++++++++++--------
>  3 files changed, 49 insertions(+), 11 deletions(-)
>
>
> proxmox:
>
> Shannon Sterz (1):
>   pve-api-types: expose certificates info endpoint
>
>  pve-api-types/Cargo.toml            |  1 +
>  pve-api-types/generate.pl           |  3 +++
>  pve-api-types/src/generated/code.rs | 15 ++++++++++++++-
>  pve-api-types/src/types/mod.rs      |  1 +
>  4 files changed, 19 insertions(+), 1 deletion(-)
>
>
> proxmox-datacenter-manager:
>
> Shannon Sterz (11):
>   client: allow users to update a changed fingerprint interactively
>   cli/api-types: move Fingerprint to common api type crate
>   server: connection: report mismatching fingerprint as untrusted on
>     probe
>   ui: wizzard: add context if a provided fingerprint did not match
>     remote
>   ui: wizzard: nodes page: always update fingerprints on user
>     confirmation
>   pdm-api-types: implement ApiType for Fingerprint
>   pdm-api-types: add staged_fingerprints field to NodeUrl
>   server: remotes: lock remotes config when updating it
>   server: connection: rotate in staged fingerprints when encountering
>     them
>   server: api: tasks: move `spawn_aborted_on_shutdown()` to super module
>   server: bin: api: tasks: add task to discover new staged certificates
>
>  cli/client/src/env/fingerprint_cache.rs       |  90 +--------
>  cli/client/src/env/mod.rs                     |   2 +-
>  lib/pdm-api-types/Cargo.toml                  |   1 +
>  lib/pdm-api-types/src/fingerprint.rs          |  84 +++++++++
>  lib/pdm-api-types/src/lib.rs                  |   3 +
>  lib/pdm-api-types/src/remotes.rs              |  15 +-
>  server/src/api/pbs/mod.rs                     |   2 +
>  server/src/api/pve/mod.rs                     |   3 +
>  server/src/api/remotes/mod.rs                 |  28 ++-
>  server/src/bin/proxmox-datacenter-api/main.rs |   1 +
>  .../tasks/ceph_detection.rs                   |  18 +-
>  .../bin/proxmox-datacenter-api/tasks/mod.rs   |  17 ++
>  .../tasks/remote_staged_fingerprints.rs       | 149 +++++++++++++++
>  server/src/connection.rs                      | 173 +++++++++++++++---
>  ui/src/remotes/config.rs                      |   1 +
>  ui/src/remotes/node_url_list.rs               |   1 +
>  ui/src/remotes/wizard_page_connect.rs         |  26 ++-
>  ui/src/remotes/wizard_page_info.rs            |   1 +
>  ui/src/remotes/wizard_page_nodes.rs           |  40 +++-
>  19 files changed, 523 insertions(+), 132 deletions(-)
>  create mode 100644 lib/pdm-api-types/src/fingerprint.rs
>  create mode 100644 server/src/bin/proxmox-datacenter-api/tasks/remote_staged_fingerprints.rs
>
>
> Summary over all repositories:
>   27 files changed, 593 insertions(+), 146 deletions(-)