From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 65CC41FF0ED for ; Fri, 31 Jul 2026 11:17:23 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 992DB214AF; Fri, 31 Jul 2026 11:17:07 +0200 (CEST) From: Shannon Sterz To: pdm-devel@lists.proxmox.com Subject: [PATCH cluster/datacenter-manager/manager/proxmox 00/15] TLS Certificate Staging Date: Fri, 31 Jul 2026 11:16:40 +0200 Message-ID: <20260731091655.93282-1-s.sterz@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1785489407604 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.158 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_LOW -0.7 Sender listed at https://www.dnswl.org/, low trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: Q62U43SAE6K2EHATZOJBS6U63WBB55UL X-Message-ID-Hash: Q62U43SAE6K2EHATZOJBS6U63WBB55UL X-MailFrom: s.sterz@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Datacenter Manager development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: the aim of this series is to allow clients to automatically adapt to regular certificate rotation. the top-level overview of the mechanism proposed here is as follows: - hosts that rotate their certificate create a new certificate at the earliest four weeks before their current certificate expires. this certificate is considered as "staged" up until it becomes actively used. - clients can query a host for a staged certificate at any moment, the host will provide information such as the fingerprint for the active and staged certificate(s). - at the earliest two weeks before their current certificate expires, hosts may start using the "staged" certificate. the two week window is needed to give clients enough time to query a potential staged certificate. - clients, that use fingerprints to validate a TLS certificate, should discard the previously used fingerprint and update to the new certificate's fingerprint (the previously staged certificate) as soon as they detect its usage. connections trying to authenticate themselves with the old Certificate should be rejected at this point. this series implements the host part of this mechanism for pve 9 and pdm. the first three patches in the series are intended for pve and implement the staging mechanism. they also make it a little easier to query the certificate of a node when we don't know the node name specifically. the next few patches improve how fingerprints are handled for pdm. they also add the certificate info endpoint to the pve client. specifically the following improvements are provided: * if a fingerprint of a remote does not match, but pdm-client is in interactive mode, allow a user to accept the updated fingerprint then and there. this better matches the behaviour in interactive mode of connecting to a non-trusted node (patch 5). * report mismatching fingerprints as untrusted when probing a remote and improve how the ui handles such situations by adding more context (patches 7-9) the remaining patches mostly prepare and then implement the rotation mechanism within pdm. pdm will query pve remote nodes once every twelve hours to see if a new staged certificate becomes available. if a new fingerprint is encountered, it will be stored in the remotes.cfg. once a staged fingerprint is encountered, it will replace the active fingerprint. How to Test ----------- the easiest way is probably to force pve to rotate and stage certificates by setting a date with `date --set` that's far enough in the future to trigger the action and then running `pveupdate`. to force pdm to query its remotes, it's easiest to run `systemctl restart proxmox-datacenter-api.service`. the daemon will execute the task query its remotes once on start. How to Apply & Bump ------------------- the first patch for pve-manager (02/15) depends on the changes for pve-cluster (01/15). the second pve-manager patch can be applied independently. note that Elias' has send a patches [1] that would address the same issue as patch (03/15), so that patch can be dropped in case elias's series makes it in first. the patches for proxmox-datacenter-manager can all be applied independently, with the exception of the last one (15/15), which needs the patch for the pve-api-types (04/15) to be applied and bumped. Future Work ----------- 1. pbs remotes currently do not rotate their certificates. a series that is as of yet not applied would add such a mechanism to pbs too [1]. for now pbs remotes are ignored by the staged certificates mechanism for the most part. 2. the `Fingerprint` type should be replaced by one from a shared proxmox-* crate. Dominik's series for unifying tls callbacks adds such a type to proxmox-http [2]. i'll adapt this series depending on how things are applied. 3. backporting of the pve patches to the bookworm branch probably makes sense to improve compatibility. i'll send such patches once this series is. this may have been more prescient when this series was first submitted. since pve 8 is eol by now (or tomorrow as of sending this), this may no longer apply. 4. somewhat orthogonal to this series: the mechanism outlined in the notes of patch 16 would probably improve adding tasks to pdm. Changelog --------- * rfc: https://lore.proxmox.com/all/20260611120327.257523-1-s.sterz@proxmox.com/ changes since the rfc: + dropped a patch adjust the tls verify callback in proxmox-client. it should instead be replaced by Dominik's implementation of a unified callback once that's applied + dropped a similar patch for proxmox-datacenter-client for the same reason. + fixed an issues where a certificate chain could be validated incorrectly + rebased on current master for all repos [1]: https://lore.proxmox.com/all/20260714145027.53038-1-e.huhsovitz@proxmox.com/ [2]: https://lore.proxmox.com/all/20260730133158.418015-1-s.sterz@proxmox.com/ [3]: https://lore.proxmox.com/all/20260701103120.1593265-6-d.csapak@proxmox.com/ pve-cluster: Shannon Sterz (1): setup: allow caller to provide the certificate filename src/PVE/Cluster/Setup.pm | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) pve-manager: Shannon Sterz (2): bin/api: add a new staged certificate when renewing self-signed cert api: certificates: if node parameter is 'localhost' return local certs PVE/API2/Certificates.pm | 10 +++++++-- PVE/CertHelpers.pm | 6 ++++++ bin/pveupdate | 44 ++++++++++++++++++++++++++++++++-------- 3 files changed, 49 insertions(+), 11 deletions(-) proxmox: Shannon Sterz (1): pve-api-types: expose certificates info endpoint pve-api-types/Cargo.toml | 1 + pve-api-types/generate.pl | 3 +++ pve-api-types/src/generated/code.rs | 15 ++++++++++++++- pve-api-types/src/types/mod.rs | 1 + 4 files changed, 19 insertions(+), 1 deletion(-) proxmox-datacenter-manager: Shannon Sterz (11): client: allow users to update a changed fingerprint interactively cli/api-types: move Fingerprint to common api type crate server: connection: report mismatching fingerprint as untrusted on probe ui: wizzard: add context if a provided fingerprint did not match remote ui: wizzard: nodes page: always update fingerprints on user confirmation pdm-api-types: implement ApiType for Fingerprint pdm-api-types: add staged_fingerprints field to NodeUrl server: remotes: lock remotes config when updating it server: connection: rotate in staged fingerprints when encountering them server: api: tasks: move `spawn_aborted_on_shutdown()` to super module server: bin: api: tasks: add task to discover new staged certificates cli/client/src/env/fingerprint_cache.rs | 90 +-------- cli/client/src/env/mod.rs | 2 +- lib/pdm-api-types/Cargo.toml | 1 + lib/pdm-api-types/src/fingerprint.rs | 84 +++++++++ lib/pdm-api-types/src/lib.rs | 3 + lib/pdm-api-types/src/remotes.rs | 15 +- server/src/api/pbs/mod.rs | 2 + server/src/api/pve/mod.rs | 3 + server/src/api/remotes/mod.rs | 28 ++- server/src/bin/proxmox-datacenter-api/main.rs | 1 + .../tasks/ceph_detection.rs | 18 +- .../bin/proxmox-datacenter-api/tasks/mod.rs | 17 ++ .../tasks/remote_staged_fingerprints.rs | 149 +++++++++++++++ server/src/connection.rs | 173 +++++++++++++++--- ui/src/remotes/config.rs | 1 + ui/src/remotes/node_url_list.rs | 1 + ui/src/remotes/wizard_page_connect.rs | 26 ++- ui/src/remotes/wizard_page_info.rs | 1 + ui/src/remotes/wizard_page_nodes.rs | 40 +++- 19 files changed, 523 insertions(+), 132 deletions(-) create mode 100644 lib/pdm-api-types/src/fingerprint.rs create mode 100644 server/src/bin/proxmox-datacenter-api/tasks/remote_staged_fingerprints.rs Summary over all repositories: 27 files changed, 593 insertions(+), 146 deletions(-) -- Generated by murpp 0.12.0