From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id B23F21FF09C for ; Mon, 19 Oct 2026 10:24:20 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id DCF28215F0; Mon, 19 Oct 2026 10:23:57 +0200 (CEST) From: Kefu Chai To: pve-devel@lists.proxmox.com Subject: [PATCH manager v2 0/4] cephx: migrate keys with a monitor this cluster does not manage Date: Mon, 19 Oct 2026 16:23:42 +0800 Message-ID: <20261019082346.1781960-1-k.chai@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1792398229340 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.282 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: KFNDTS7PEEV3QDRNFSXT2NCABU3ABGOH X-Message-ID-Hash: KFNDTS7PEEV3QDRNFSXT2NCABU3ABGOH X-MailFrom: k.chai@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Thomas Lamprecht , Jonas Theisen X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hi everyone, A user reported an interesting stretch-cluster setup that pve-cephx-rotate-service-keys wasn't ready for: a tiebreaker monitor running on a standalone Proxmox VE host that isn't a node of the cluster itself, joined to the corosync quorum only through a QDevice and never listed under /etc/pve/nodes. All five monitors were in quorum and on the same, capable Ceph version, but the dry run still failed twice over: FAIL: could not reach node 'ch95o-pvewp01': command failed on node 'ch95o-pvewp01': command '/usr/bin/ssh ... perl - ceph mon:ch95o-pvewp01' failed: exit code 255 could not verify the installed Ceph version of mon. on node 'ch95o-pvewp01'; Both failures trace back to the same two assumptions: that a daemon can always be reached with a shell on its host, and that pvestatd on some node of this cluster always knows what's installed there. Neither holds for a monitor we don't manage, so this series walks through getting the helper past both, step by step. First, we teach it to ask a monitor about itself without needing a shell at all: over its admin socket where the helper already has a route, and over 'ceph tell mon.' otherwise. That alone covers the three questions behind the ssh failure above (auth methods required, pending-key support, and current sessions). With the monitor reachable again, the version check is next: rather than insisting on an installed version from pvestatd, which only nodes of this cluster report, we judge such a daemon by the version it runs, since Ceph already reports that for every daemon in the cluster regardless of who manages it. That clears the dry run, so the third step is making the actual rotation work: the 'mon.' key change itself writes to the auth database and reaches every monitor through paxos, so an unmanaged monitor gets migrated right along with the others. Only refreshing its emergency keyring still needs a shell it doesn't have, and the helper now says so plainly instead of refusing the whole rotation. Last, since that emergency keyring (and any host's own 'client.admin' keyring) is left stale by design, we make sure the helper tells the operator exactly which hosts and files need a manual refresh, and how. To check all this against a real cluster and not just the test suite, I built a nested lab matching the reported shape: three Proxmox VE nodes running as VMs, clustered with Ceph and a monitor each on the first two, and a fourth VM running only ceph-mon, joined straight to the Ceph cluster without ever joining the Proxmox VE cluster, playing the tiebreaker. Running the patched helper there end to end, from dry run through the monitor-key rotation to the final client-key notes, matched what the series is meant to do. Together these four steps let the reported cluster's OSD, manager, MDS and monitor keys rotate with the tiebreaker happily in place. 'client.admin' and per-storage keys still need a manual procedure for now, that's a separate piece of work I'm looking at next. Changes since v1: - Patch 1: skip the 'mon.' session that 'ceph tell' itself adds to a monitor's session list. On a cluster that keeps 'mon.' out of the auth database, it could have blocked the run. - Patch 4: fix the command printed for refreshing a staged key. v1 suggested saving the output of 'auth get', but that file has a 'pending key' line which Ceph cannot read back. I only noticed this when trying it on the lab, sorry about that. The command now takes the key from the JSON output, and runs as 'client.admin', since a user with only 'profile rbd' caps cannot read its own key. - Some small cleanups, and shorter comments and commit messages. - Rebased on current master. I ran the series again on the same lab, including staging and committing the 'client.admin' key with the external monitor in place. v1: https://lore.proxmox.com/pve-devel/20260927005906.4184138-1-k.chai@proxmox.com/ As always, happy to hear any thoughts or concerns, thanks for reading! Kefu Chai (4): migrations: cephx: ask a monitor about itself through one route migrations: cephx: judge cipher support by the version a daemon runs migrations: cephx: rotate the monitor key with monitors we do not manage migrations: cephx: name the hosts keeping key copies we cannot write PVE/Ceph/KeyMigration.pm | 21 +- bin/pve-cephx-rotate-service-keys | 251 ++++++++++++++----- test/CephKeyMigrationScript_test.pl | 358 +++++++++++++++++++++++++++- test/CephKeyMigration_test.pl | 28 +++ 4 files changed, 586 insertions(+), 72 deletions(-) -- 2.47.3