From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 8CF6E1FF13C for ; Thu, 11 Jun 2026 14:04:15 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 6E1C53EE3; Thu, 11 Jun 2026 14:04:15 +0200 (CEST) From: Shannon Sterz To: pdm-devel@lists.proxmox.com Subject: [RFC cluster/datacenter-manager/manager/proxmox 00/17] TLS Certificate Staging Date: Thu, 11 Jun 2026 14:03:10 +0200 Message-ID: <20260611120327.257523-1-s.sterz@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1781179363706 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.109 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 46OEGUNX34FOFRBAY3DBUMLYJTQDLKNL X-Message-ID-Hash: 46OEGUNX34FOFRBAY3DBUMLYJTQDLKNL X-MailFrom: s.sterz@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Datacenter Manager development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: the aim of this series is to allow clients to automatically adapt to regular certificate rotation. the top-level overview of the mechanism proposed here is as follows: - hosts that rotate their certificate create a new certificate at the earliest four weeks before their current certificate expires. this certificate is considered as "staged" up until it becomes actively used. - clients can query a host for a staged certificate at any moment, the host will provide information such as the fingerprint for the active and staged certificate(s). - at the earliest two weeks before their current certificate expires, hosts may start using the "staged" certificate. the two week window is needed to give clients enough time to query a potential staged certificate. - clients, that use fingerprints to validate a TLS certificate, should discard the previously used fingerprint and update to the new certificate's fingerprint (the previously staged certificate) as soon as they detect its usage. connections trying to authenticate themselves with the old Certificate should be rejected at this point. this series implements the host part of this mechanism for pve 9 and pdm. the first three patches in the series are intended for pve and implement the staging mechanism. they also make it a little easier to query the certificate of a node when we don't know the node name specifically. the next few patches improve how fingerprints are handled for the proxmox-client and pdm specifically. they also add the certificate info endpoint to the pve client. specifically the following improvements are provided: * if a fingerprint is provided, never fall back to the system's trust store. providing proper pinning semantics (patches 4 and 6) * if a fingerprint of a remote does not match, but pdm-client is in interactive mode, allow a user to accept the updated fingerprint then and there. this better matches the behaviour in interactive mode of connecting to a non-trusted node (patch 7). * report mismatching fingerprints as untrusted when probing a remote and improve how the ui handles such situations by adding more context (patches 9-11) the remaining commits mostly prepare and the implement the rotation mechanism within pdm. pdm will query pve remote nodes once every twelve hours to see if a new staged certificate becomes available. if a new fingerprint is encountered, it will be stored in the remotes.cfg. once a staged fingerprint is encountered, it will replace the active fingerprint. How to Test ----------- the easiest way is probably to force pve to rotate and stage certificates by setting a date with `date --set` that's far enough in the future to trigger the action and then running `pveupdate`. to force pdm to query its remotes, it's easiest to run `systemctl restart proxmox-datacenter-api.service`. the daemon will execute the task query its remotes once on start. How to Apply & Bump ------------------- the first patch for pve-manager (02/17) depends on the changes for pve-cluster (01/17). the second pve-manager patch can be applied independently. the patches for proxmox-datacenter-manager can all be applied independently, with the exception of the last one (17/17), which needs the patch for the pve-api-types (05/17) to be applied and bumped. Future Work ----------- 1. pbs remotes currently do not rotate their certificates. a series that is as of yet not applied would add such a mechanism to pbs too. for now pbs remotes are ignored by the staged certificates mechanism for the most part. 2. backporting of the pve patches to the bookworm branch probably makes sense to improve compatibility. i'll send such patches once this series is. 3. somewhat orthogonal to this series: the mechanism outlined in the notes of patch 16 would probably improve adding tasks to pdm. pve-cluster: Shannon Sterz (1): setup: allow caller to provide the certificate filename src/PVE/Cluster/Setup.pm | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) pve-manager: Shannon Sterz (2): bin/api: add a new staged certificate when renewing self-signed cert api: certificates: if node parameter is 'localhost' return local certs PVE/API2/Certificates.pm | 10 +++++++-- PVE/CertHelpers.pm | 6 ++++++ bin/pveupdate | 44 ++++++++++++++++++++++++++++++++-------- 3 files changed, 49 insertions(+), 11 deletions(-) proxmox: Shannon Sterz (2): client: ignore certificate trust store validation result on fp option pve-api-types: expose certificates info endpoint proxmox-client/src/client.rs | 5 +---- pve-api-types/Cargo.toml | 1 + pve-api-types/generate.pl | 3 +++ pve-api-types/src/generated/code.rs | 15 ++++++++++++++- pve-api-types/src/types/mod.rs | 1 + 5 files changed, 20 insertions(+), 5 deletions(-) proxmox-datacenter-manager: Shannon Sterz (12): client: don't short-circuit on valid certificate when tls fp exists client: allow users to update a changed fingerprint interactively cli/api-types: move Fingerprint to common api type crate server: connection: report mismatching fingerprint as untrusted on probe ui: wizzard: add context if a provided fingerprint did not match remote ui: wizzard: nodes page: always update fingerprints on user confirmation pdm-api-types: implement ApiType for Fingerprint pdm-api-types: add staged_fingerprints field to NodeUrl server: remotes: lock remotes config when updating it server: connection: rotate in staged fingerprints when encountering them server: api: tasks: move `spawn_aborted_on_shutdown()` to super module server: bin: api: tasks: add task to discover new staged certificates cli/client/src/env/fingerprint_cache.rs | 91 ++-------- cli/client/src/env/mod.rs | 10 +- cli/client/src/main.rs | 6 +- lib/pdm-api-types/Cargo.toml | 1 + lib/pdm-api-types/src/fingerprint.rs | 84 +++++++++ lib/pdm-api-types/src/lib.rs | 3 + lib/pdm-api-types/src/remotes.rs | 15 +- server/src/api/pbs/mod.rs | 2 + server/src/api/pve/mod.rs | 3 + server/src/api/remotes/mod.rs | 28 ++- server/src/bin/proxmox-datacenter-api/main.rs | 1 + .../tasks/ceph_detection.rs | 18 +- .../bin/proxmox-datacenter-api/tasks/mod.rs | 17 ++ .../tasks/remote_staged_fingerprints.rs | 149 ++++++++++++++++ server/src/connection.rs | 166 +++++++++++++++--- ui/src/remotes/config.rs | 1 + ui/src/remotes/node_url_list.rs | 1 + ui/src/remotes/wizard_page_connect.rs | 26 ++- ui/src/remotes/wizard_page_info.rs | 1 + ui/src/remotes/wizard_page_nodes.rs | 40 ++++- 20 files changed, 525 insertions(+), 138 deletions(-) create mode 100644 lib/pdm-api-types/src/fingerprint.rs create mode 100644 server/src/bin/proxmox-datacenter-api/tasks/remote_staged_fingerprints.rs Summary over all repositories: 29 files changed, 596 insertions(+), 156 deletions(-) -- Generated by murpp 0.10.0