[PATCH manager 0/3] migrations: cephx: catch daemons whose key on disk is stale

[Kefu Chai <k.chai@proxmox.com>]

A user ran into the following after the upgrade to Ceph 20: an admin
rotated the keys of all MDS daemons by hand with
'ceph auth rotate --key-type=aes256k', then ran the migration helper.
The helper only looked at the cipher of each key in the auth database,
saw aes256k, and reported PASS. The keyrings on disk still held the old
keys, so when the MDS daemons restarted they failed with
handle_auth_bad_method and CephFS went down. This series is my attempt
to catch that case.

Patch 1 moves the places that collect a run's cluster information into
a single helper. While doing so I found some older bugs it fixes: the
RADOS handle got lost after resuming a lockbox key, the preferred
cipher was overwritten by a later read from the monitor map, which
reports an unsupported value as 'unknown', and daemons recovered from a
saved plan were dropped again after the client key steps. It does not
depend on the other patches, so it could go in on its own.

Patch 2 compares the key each manager, metadata server and OSD reads at
its next start with the key the monitors hold. That is the key in its
keyring and, for an OSD, also the one in its bluestore label. It warns
about a daemon whose copy differs, and if a copy differs or cannot be
compared, the verdicts that report the cluster as migrated become a
warning naming that daemon instead of a PASS. The cost is one more SSH
session per node on every run, dry runs included.

Patch 3 then plans such a daemon for rotation like one still on the old
cipher, so the helper fixes the copies itself. I split it out in case
you would rather take only the detection, or want to discuss the
rotation part separately.

Besides the unit and script tests, I tried it on a small Proxmox VE 9.2
cluster with Ceph 20.2.4. With the final version:
- an MDS rotated by hand and stopped, with '--restart-daemons': the
  plan named its start command, its key was written, and it started
  fine when I started it by hand afterwards
- a running MDS rotated by hand while another node was unreachable: its
  key was swapped live, and the closing verdict was a warning naming
  the daemons on the unreachable node
With an earlier version, which differed in messages and checks but not
in how keys are rotated:
- a running MDS rotated by hand: its key was swapped live, and it
  restarted cleanly afterwards
- an OSD rotated by hand: its label and keyring were both rewritten,
  and it restarted cleanly
- a '--restart-daemons' run interrupted with SIGINT halfway: the next
  run planned only the daemon it had not reached

One thing I was unsure about: with a stale key on disk, a dry run still
prints "Preflight checks passed for the displayed plan", as for any
other plan. If that reads too much like a PASS for the cluster, I am
happy to change it.

Kefu Chai (3):
  migrations: cephx: collect the run's cluster information in one place
  migrations: cephx: report daemons whose key on disk differs
  migrations: cephx: rotate daemons whose key on disk differs

 PVE/Ceph/KeyMigration.pm            |  68 ++++-
 bin/pve-cephx-rotate-service-keys   | 230 +++++++++++++----
 test/CephKeyMigrationScript_test.pl | 380 ++++++++++++++++++++++++++++
 test/CephKeyMigration_test.pl       | 138 ++++++++++
 4 files changed, 766 insertions(+), 50 deletions(-)

-- 
2.47.3