From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 087581FF129 for ; Thu, 06 Aug 2026 17:41:21 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id C982421501; Thu, 06 Aug 2026 17:41:20 +0200 (CEST) Date: Thu, 06 Aug 2026 17:41:10 +0200 From: Fabian =?iso-8859-1?q?Gr=FCnbichler?= Subject: Re: [PATCH cluster/common/datacenter-manager/manager/proxmox v2 00/16] TLS Certificate Staging To: pdm-devel@lists.proxmox.com, Shannon Sterz References: <20260805131838.254723-2-s.sterz@proxmox.com> In-Reply-To: <20260805131838.254723-2-s.sterz@proxmox.com> MIME-Version: 1.0 User-Agent: astroid/0.17.0 (https://github.com/astroidmail/astroid) Message-Id: <1786030751.td161ru3ry.astroid@yuna.none> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1786030858913 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.091 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_LOW -0.7 Sender listed at https://www.dnswl.org/, low trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: GJ7F7PGKLSWZDIHWXMV6VNYPAENWENG2 X-Message-ID-Hash: GJ7F7PGKLSWZDIHWXMV6VNYPAENWENG2 X-MailFrom: f.gruenbichler@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Datacenter Manager development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On August 5, 2026 3:18 pm, Shannon Sterz wrote: > the aim of this series is to allow clients to automatically adapt to regu= lar > certificate rotation. the top-level overview of the mechanism proposed he= re is > as follows: I like the principal approach of this. left some comments that are hopefully actionable. we should also do the same thing for PBS, which means we also need handling code for staging fingerprints in the PBS client(s). I think most of the perl certificate handling code can probably move to the proxmox-tls-certificates crate you want to introduce. if that exposes the right set of helpers, then we hopefully only have a single code base for all of the lower level certificate handling (at some point). > - hosts that rotate their certificate create a new certificate at the ear= liest > four weeks before their current certificate expires. this certificate i= s > considered as "staged" up until it becomes actively used. > - clients can query a host for a staged certificate at any moment, the ho= st > will provide information such as the fingerprint for the active and sta= ged > certificate(s). > - at the earliest two weeks before their current certificate expires, hos= ts may > start using the "staged" certificate. the two week window is needed to = give > clients enough time to query a potential staged certificate. > - clients, that use fingerprints to validate a TLS certificate, should di= scard > the previously used fingerprint and update to the new certificate's > fingerprint (the previously staged certificate) as soon as they detect = its > usage. connections trying to authenticate themselves with the old certi= ficate > should be rejected at this point. >=20 > this series implements the host part of this mechanism for pve 9 and pdm.= the > first three patches in the series are intended for pve and implement the > staging mechanism. they also make it a little easier to query the certifi= cate > of a node when we don't know the node name specifically. >=20 > the next few patches improve how fingerprints are handled for pdm. they a= lso > add the certificate info endpoint to the pve client. specifically the fol= lowing > improvements are provided: >=20 > * if a fingerprint of a remote does not match, but pdm-client is in inter= active > mode, allow a user to accept the updated fingerprint then and there. th= is > better matches the behaviour in interactive mode of connecting to a > non-trusted node (patch 6). > * report mismatching fingerprints as untrusted when probing a remote and > improve how the ui handles such situations by adding more context (patc= hes > 8-10) >=20 > the remaining patches mostly prepare and then implement the rotation mech= anism > within pdm. pdm will query pve remote nodes once every twelve hours to se= e if a > new staged certificate becomes available. if a new fingerprint is encount= ered, > it will be stored in the remotes.cfg. once a staged fingerprint is encoun= tered, > it will replace the active fingerprint. >=20 > How to Test > ----------- >=20 > the easiest way is probably to force pve to rotate and stage certificates= by > setting a date with `date --set` that's far enough in the future to trigg= er the > action and then running `pveupdate`. to force pdm to query its remotes, i= t's > easiest to run `systemctl restart proxmox-datacenter-api.service`. the da= emon > will execute the task query its remotes once on start. >=20 > How to Apply & Bump > ------------------- >=20 > the first patch for pve-manager (03/16) depends on the changes for pve-cl= uster > (01/16) and pve-common (02/16). the second pve-manager patch can be appli= ed > independently. note that Elias has sent patches [1] that would address th= e same > issue as patch (04/16), so that patch can be dropped in case Elias' serie= s > makes it in first. >=20 > the patches for proxmox-datacenter-manager can all be applied independent= ly, > with the exception of the last one (16/16), which needs the patch for the > pve-api-types (05/16) to be applied and bumped. >=20 > Future Work > ----------- >=20 > 1. pbs remotes currently do not rotate their certificates. a series that = is as > of yet not applied would add such a mechanism to pbs too [2]. for now pbs > remotes are ignored by the staged certificates mechanism for the most par= t. >=20 > 2. the `Fingerprint` type should be replaced by one from a shared proxmox= -* > crate. Dominik's series for unifying tls callbacks adds such a type to > proxmox-http [3]. i'll adapt this series depending on how things are appl= ied. >=20 > 3. backporting of the pve patches to the bookworm branch probably makes s= ense > to improve compatibility. i'll send such patches once this series is appr= oved. > this may have been more prescient when this series was first submitted. s= ince > pve 8 is eol by now (or tomorrow as of sending this), this may no longer = apply. >=20 > 4. somewhat orthogonal to this series: the mechanism outlined in the note= s of > patch 15 would probably improve adding tasks to pdm. talked about this wi= th > Lukas a little and he agrees. possibly in combination with a message pass= ing > mechanism. >=20 > 5. i'll add documentation to this series once the general mechanism is > deemed appropriate or if it's applied earlier in a follow-up. >=20 > Changelog > --------- >=20 > * v1: https://lore.proxmox.com/all/20260731091655.93282-1-s.sterz@proxmox= .com/ >=20 > changes since v1: >=20 > + always return leaf certificate fingerprint when probing remotes > + clean up an issue where the staged certificate could be regenerated= once > a day during the staging period > + clean up usages of `tokio::spawn` that did blocking io with > `tokio::task::spawn_blocking` > + fix an issue where the ca could have been rotated between staging a > certificate and using it. > + clean up stale staged certificates properly > + other smaller clean ups (more correct phrasings etc). >=20 > * rfc: https://lore.proxmox.com/all/20260611120327.257523-1-s.sterz@proxm= ox.com/ >=20 > changes since the rfc: >=20 > + dropped a patch adjusting the tls verify callback in proxmox-client= . it > should instead be replaced by Dominik's implementation of a unified > callback once that's applied > + dropped a similar patch for proxmox-datacenter-client for the same > reason. > + fixed an issue where a certificate chain could be validated incorre= ctly > + rebased on current master for all repos >=20 > [1]: https://lore.proxmox.com/all/20260714145027.53038-1-e.huhsovitz@prox= mox.com/ > [2]: https://lore.proxmox.com/all/20260730133158.418015-1-s.sterz@proxmox= .com/ > [3]: https://lore.proxmox.com/all/20260701103120.1593265-6-d.csapak@proxm= ox.com/ >=20 >=20 > pve-cluster: >=20 > Shannon Sterz (1): > setup: allow caller to provide the certificate filename >=20 > src/PVE/Cluster/Setup.pm | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) >=20 >=20 > pve-common: >=20 > Shannon Sterz (1): > certificate: add helper to verify that a certificate was signed by a > ca >=20 > src/PVE/Certificate.pm | 46 ++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 46 insertions(+) >=20 >=20 > pve-manager: >=20 > Shannon Sterz (2): > bin/api: add a new staged certificate when renewing self-signed cert > api: certificates: if node parameter is 'localhost' return local certs >=20 > PVE/API2/Certificates.pm | 10 ++++-- > PVE/CertHelpers.pm | 6 ++++ > bin/pveupdate | 72 +++++++++++++++++++++++++++++++++------- > 3 files changed, 74 insertions(+), 14 deletions(-) >=20 >=20 > proxmox: >=20 > Shannon Sterz (1): > pve-api-types: expose certificates info endpoint >=20 > pve-api-types/Cargo.toml | 1 + > pve-api-types/debian/control | 2 ++ > pve-api-types/generate.pl | 3 +++ > pve-api-types/src/generated/code.rs | 15 ++++++++++++++- > pve-api-types/src/types/mod.rs | 1 + > 5 files changed, 21 insertions(+), 1 deletion(-) >=20 >=20 > proxmox-datacenter-manager: >=20 > Shannon Sterz (11): > client: allow users to update a changed fingerprint interactively > cli/api-types: move Fingerprint to common api type crate > server: connection: report mismatching fingerprint as untrusted on > probe > ui: wizard: add context if a provided fingerprint did not match remote > ui: wizard: nodes page: always update fingerprints on user > confirmation > pdm-api-types: implement ApiType for Fingerprint > pdm-api-types: add staged_fingerprints field to NodeUrl > server: remotes: lock remotes config when updating it > server: connection: rotate in staged fingerprints when encountering > them > server: api: tasks: move `spawn_aborted_on_shutdown()` to super module > server: bin: api: tasks: add task to discover new staged certificates >=20 > cli/client/src/env/fingerprint_cache.rs | 90 +-------- > cli/client/src/env/mod.rs | 2 +- > lib/pdm-api-types/Cargo.toml | 1 + > lib/pdm-api-types/src/fingerprint.rs | 84 ++++++++ > lib/pdm-api-types/src/lib.rs | 3 + > lib/pdm-api-types/src/remotes.rs | 15 +- > server/src/api/pbs/mod.rs | 2 + > server/src/api/pve/mod.rs | 3 + > server/src/api/remotes/mod.rs | 36 +++- > server/src/bin/proxmox-datacenter-api/main.rs | 1 + > .../tasks/ceph_detection.rs | 18 +- > .../bin/proxmox-datacenter-api/tasks/mod.rs | 17 ++ > .../tasks/remote_staged_fingerprints.rs | 156 +++++++++++++++ > server/src/connection.rs | 179 +++++++++++++++--- > ui/src/remotes/config.rs | 1 + > ui/src/remotes/node_url_list.rs | 1 + > ui/src/remotes/wizard_page_connect.rs | 26 ++- > ui/src/remotes/wizard_page_info.rs | 1 + > ui/src/remotes/wizard_page_nodes.rs | 40 +++- > 19 files changed, 544 insertions(+), 132 deletions(-) > create mode 100644 lib/pdm-api-types/src/fingerprint.rs > create mode 100644 server/src/bin/proxmox-datacenter-api/tasks/remote_st= aged_fingerprints.rs >=20 >=20 > Summary over all repositories: > 29 files changed, 687 insertions(+), 149 deletions(-) >=20 > --=20 > Generated by murpp 0.12.0 >=20 >=20 >=20 >=20 >=20