From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 4B7811FF09B for ; Mon, 12 Oct 2026 16:38:14 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id CC4C62156E; Mon, 12 Oct 2026 16:38:11 +0200 (CEST) From: Kefu Chai To: pve-devel@lists.proxmox.com Subject: [PATCH manager 0/3] migrations: cephx: catch daemons whose key on disk is stale Date: Mon, 12 Oct 2026 22:37:59 +0800 Message-ID: <20261012143802.1193012-1-k.chai@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1791815886873 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.368 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: YI3E5WTCIPZ7ZOJRB3DD3EYGX6Q377MM X-Message-ID-Hash: YI3E5WTCIPZ7ZOJRB3DD3EYGX6Q377MM X-MailFrom: k.chai@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: A user ran into the following after the upgrade to Ceph 20: an admin rotated the keys of all MDS daemons by hand with 'ceph auth rotate --key-type=aes256k', then ran the migration helper. The helper only looked at the cipher of each key in the auth database, saw aes256k, and reported PASS. The keyrings on disk still held the old keys, so when the MDS daemons restarted they failed with handle_auth_bad_method and CephFS went down. This series is my attempt to catch that case. Patch 1 moves the places that collect a run's cluster information into a single helper. While doing so I found some older bugs it fixes: the RADOS handle got lost after resuming a lockbox key, the preferred cipher was overwritten by a later read from the monitor map, which reports an unsupported value as 'unknown', and daemons recovered from a saved plan were dropped again after the client key steps. It does not depend on the other patches, so it could go in on its own. Patch 2 compares the key each manager, metadata server and OSD reads at its next start with the key the monitors hold. That is the key in its keyring and, for an OSD, also the one in its bluestore label. It warns about a daemon whose copy differs, and if a copy differs or cannot be compared, the verdicts that report the cluster as migrated become a warning naming that daemon instead of a PASS. The cost is one more SSH session per node on every run, dry runs included. Patch 3 then plans such a daemon for rotation like one still on the old cipher, so the helper fixes the copies itself. I split it out in case you would rather take only the detection, or want to discuss the rotation part separately. Besides the unit and script tests, I tried it on a small Proxmox VE 9.2 cluster with Ceph 20.2.4. With the final version: - an MDS rotated by hand and stopped, with '--restart-daemons': the plan named its start command, its key was written, and it started fine when I started it by hand afterwards - a running MDS rotated by hand while another node was unreachable: its key was swapped live, and the closing verdict was a warning naming the daemons on the unreachable node With an earlier version, which differed in messages and checks but not in how keys are rotated: - a running MDS rotated by hand: its key was swapped live, and it restarted cleanly afterwards - an OSD rotated by hand: its label and keyring were both rewritten, and it restarted cleanly - a '--restart-daemons' run interrupted with SIGINT halfway: the next run planned only the daemon it had not reached One thing I was unsure about: with a stale key on disk, a dry run still prints "Preflight checks passed for the displayed plan", as for any other plan. If that reads too much like a PASS for the cluster, I am happy to change it. Kefu Chai (3): migrations: cephx: collect the run's cluster information in one place migrations: cephx: report daemons whose key on disk differs migrations: cephx: rotate daemons whose key on disk differs PVE/Ceph/KeyMigration.pm | 68 ++++- bin/pve-cephx-rotate-service-keys | 230 +++++++++++++---- test/CephKeyMigrationScript_test.pl | 380 ++++++++++++++++++++++++++++ test/CephKeyMigration_test.pl | 138 ++++++++++ 4 files changed, 766 insertions(+), 50 deletions(-) -- 2.47.3