From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id C5AAD1FF0B2 for ; Sun, 25 Oct 2026 10:57:51 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 60C2421451; Sun, 25 Oct 2026 10:57:42 +0100 (CET) From: Kefu Chai To: pve-devel@lists.proxmox.com Subject: [PATCH manager v2 0/3] migrations: cephx: catch daemons whose key on disk is stale Date: Sun, 25 Oct 2026 17:57:27 +0800 Message-ID: <20261025095730.4027539-1-k.chai@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1792922254539 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.283 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: MEAS7ZLQW344MB2FQ53J2374ZWGKPBGC X-Message-ID-Hash: MEAS7ZLQW344MB2FQ53J2374ZWGKPBGC X-MailFrom: k.chai@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hi, after upgrading to Ceph 20, a user rotated their MDS keys by hand with 'ceph auth rotate --key-type=aes256k' and then ran the migration helper. It saw aes256k in the auth database and happily said PASS, but the keyrings on disk still had the old keys. So the MDS daemons couldn't authenticate at their next restart, and CephFS went down. Since v1 we've also seen this with OSDs, and there fixing the keyring isn't even enough, because ceph-volume rebuilds it from the bluestore label at every boot. With this series the helper also looks at the keys the daemons will actually read next time, i.e. the keyrings and the OSD labels. Any daemon with a stale copy gets planned for rotation, which rewrites all its copies, and if a copy can't be checked, the helper won't call the cluster migrated. The cost is one more SSH session per node per run. Patch 1 is a cleanup that fixes a few older bugs on the way, so feel free to pick it up on its own. I gave it a spin on a three node cluster with Ceph 20.2.4, with a stale OSD label: with the OSD running, with '--restart-daemons', and with it down at its start limit. Each time the OSD came back fine after a 'ceph-volume lvm activate', which is what a reboot does. One thing I'm unsure about: a dry run with a stale key still prints "Preflight checks passed for the displayed plan". If that reads too much like a PASS to you, I'm happy to change it. Changes since v1: - rebased; only daemons on cluster nodes are probed now, the rest are listed as not compared - dropped the special case for a staged pending key on disk - stale daemons are counted in needs_rotation() - the start hint for a stopped daemon runs 'systemctl reset-failed' first - the "already on the new cipher" message no longer credits an earlier run - reworked the tests to match the existing ones v1: https://lore.proxmox.com/pve-devel/20261012143802.1193012-1-k.chai@proxmox.com/ Thanks for having a look! Kefu Chai (3): migrations: cephx: collect the run's cluster information in one place migrations: cephx: report daemons whose key on disk differs migrations: cephx: rotate daemons whose key on disk differs PVE/Ceph/KeyMigration.pm | 60 +++++- bin/pve-cephx-rotate-service-keys | 249 +++++++++++++++++++------ test/CephKeyMigrationScript_test.pl | 278 ++++++++++++++++++++++++++++ test/CephKeyMigration_test.pl | 91 +++++++++ 4 files changed, 619 insertions(+), 59 deletions(-) -- 2.47.3